Return server readiness only after query serving starts

[goutamadwant]

Summary

• keep server readiness endpoints unavailable until the query-serving path is ready
• reuse the server's existing local readiness state, which becomes ready after the query server starts and IS_SHUTDOWN_IN_PROGRESS is cleared
• preserve liveness behavior and add regression coverage for the startup window

Why

During startup, ServiceStatus can become GOOD before the query server is started and before brokers can route queries back to the server. This allowed /health and /health/readiness to return 200 too early during rolling restarts.

The health resource uses the existing local readiness signal instead of reading Helix on every probe.

This addresses item 1 of #16565. The broker external-view watermark described in item 2 remains out of scope.

Testing

• ./mvnw -pl pinot-server -am -Dtest=HealthCheckResourceTest,AccessControlTest -Dsurefire.failIfNoSpecifiedTests=false test
• ./mvnw spotless:apply -pl pinot-server
• ./mvnw license:format -pl pinot-server
• ./mvnw checkstyle:check -pl pinot-server
• ./mvnw license:check -pl pinot-server
• ./mvnw test-compile -pl pinot-server -am -Dmaven.compiler.showDeprecation=true -Dmaven.compiler.showWarnings=true '-Dmaven.compiler.compilerArgs=-Xlint:all'

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 66.62%. Comparing base (d3604a5) to head (fff4fff).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
.../pinot/server/starter/helix/BaseServerStarter.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19178      +/-   ##
============================================
- Coverage     66.62%   66.62%   -0.01%     
  Complexity     1423     1423              
============================================
  Files          3443     3443              
  Lines        218577   218579       +2     
  Branches      34792    34792              
============================================
- Hits         145624   145621       -3     
- Misses        61218    61225       +7     
+ Partials      11735    11733       -2     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	66.62% <85.71%> (-0.01%)	⬇️
temurin	66.62% <85.71%> (-0.01%)	⬇️
unittests	66.61% <85.71%> (-0.01%)	⬇️
unittests1	57.12% <ø> (-0.01%)	⬇️
unittests2	38.93% <85.71%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Jackie-Jiang]

Thanks for the contribution.

Unfortunately this solution is still not good enough. There is a delay for broker to get the server config change callback and add it to the routing.
We need to find a way to ensure broker already add the server to the routing table

[goutamadwant]

Thanks @Jackie-Jiang you are right. The current change only ensures that the query server is running before shutdownInProgress is cleared. It then marks the server ready immediately, while brokers process the instance-config callback asynchronously, so readiness can still return 200 before the server has been added to their routing tables.

My proposed next step is an explicit broker-side acknowledgement or processing watermark after the routing manager finishes applying the instance-config change to all routing entries. The server would remain unready until the relevant live brokers have acknowledged the config version that re-enabled it.

Does that coordination point match the approach you prefer, or should this condition instead be exposed to the rolling-restart orchestrator rather than gating the server health endpoint? Let me know. thanks!