Repository navigation
Conversation
Replace the poisoned backend process group after a typed runtime CUDA OOM while preserving the API process and bounded new admissions. Retire all old workers and reject traffic from older epochs before opening the new generation. Constraint: Recovery defaults off and dispatched requests are never replayed. Scope-risk: moderate
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 20, 2026
FlashML-org#764 FlashML-org#763 FlashML-org#708 FlashML-org#687 deferred, FlashML-org#688 kept as a feature; FlashML-org#169 kept after its upstream close, FlashML-org#494/FlashML-org#154/FlashML-org#315 replaced upstream by FlashML-org#690/FlashML-org#698; FlashML-org#278 dropped with upstream's auto->hybrid pick (FlashML-org#734). Assisted-by: Claude Opus 5.5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runtime CUDA OOM currently terminates backend workers and leaves recovery to the outer daemon. This adds opt-in
--oom-recoveryto retire and reap the failed backend group, start a fresh group, and preserve the API process. The default remains off.Dispatched requests receive one explicit terminal error and are never replayed. New requests may wait in a bounded admission queue until replacement readiness. Restart attempts and readiness are bounded; backend epochs reject stale requests, replies, aborts and cache-rebuild controls. Shutdown takes precedence and other failures retain the daemon fallback. Raw completion responses now preserve backend error codes in both streaming and non-streaming responses.
Validation against upstream
f02fbb1, final candidatea4ebbfd:torch.cuda.OutOfMemoryError. The failed stream emittedinsufficient_workspaceand[DONE].eb9e70f; subsequent product changes only preserve raw error-code serialization. The final revision has fresh CPU coverage and the full enabled recovery test above. Source and GPU evidence received independent review before publication.Earlier validation rounds remain recorded as
failed_harness: one stopped on a transient GPU-query timeout; another exposed the missing raw-completion error code and stopped before its remaining assertions. They are not counted as passing recovery rounds. Private fault hooks and host-specific validation controls are excluded from this PR.GPU coverage is limited to one GPU and raw completions; other protocol adapters have CPU coverage. This changes OOM lifecycle recovery, not allocation sizing, prefill algorithms or cache policy, and makes no throughput or multi-GPU claim. Related: #110, #172.