Repository navigation
fix(DM01-6341): catch K8s API timeout in kube_job to prevent crash loop - #651
Merged
Merged
Conversation
When K8s API is slow (e.g. during NodeOS upgrade), requests.post() in kube_job() raises ReadTimeout. This was uncaught and propagated out of handle() since schedule() was outside the try/except block, crashing the scheduler process (95 restarts observed in 2 days). - Wrap requests.post() in kube_job() with try/except Timeout; return False so the job stays queued and is retried on the next tick - Move assign_cluster() and schedule() inside the existing try/except in handle() so no uncaught exception can escape to main()
- Broaden exception catch from Timeout to RequestException to also handle ConnectionError during K8s API unavailability - Return True on 409 in kube_job() so a job is not stuck queued when a previous POST succeeded but the response timed out - Move update_cluster_state() inside the try/except block in handle() to eliminate a remaining uncaught exception path
…ground threads handle_pipeline_invocations() and handle_function_invocations() make live K8s API calls every tick, blocking handle_orphaned_jobs() and schedule() when K8s API is slow. Move them to dedicated background threads (same pattern as nodes/pipelines cache refresh) so the main scheduling loop is never blocked by CRD controller sync latency. sleep is placed after handle() to prevent concurrent runs if a single iteration takes longer than the sleep interval.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
requests.post()inkube_job()withtry/except RequestException— catches bothTimeoutandConnectionError; returnsFalseso the job stays queued and is retried on the next tickkube_job()as success — if a previous POST succeeded but the response timed out, the next attempt gets 409; treating it as success prevents the job from being stuck in queued foreverassign_cluster(),schedule(), andupdate_cluster_state()inside the singletry/exceptinhandle()so no uncaught exception can escape tomain()Root Cause
When K8s API is slow (observed during NodeOS upgrade),
requests.post()inkube_job()raisesReadTimeout.schedule()was called outside thetry/exceptblock inhandle(), so the exception propagated uncaught tomain(), crashing the scheduler process. This caused 95 pod restarts in 2 days on the production cluster.Behavior Change
Previously, an exception in
handle_aborts()/handle_timeouts()etc. was caught andschedule()still ran. Now an exception in any part ofhandle()skipsschedule()for that tick. This is intentional: if an early function fails (typically a DB issue), attempting to schedule would also fail against the same broken connection.Test plan