fix: wait for cold-start jobs and print a runnable curl example after deploy by justinwlin · Pull Request #394 · runpod/flash

justinwlin · 2026-10-27T14:15:59Z

Note

Companion PR: runpod/docs#923. This PR fixes the cold-start failures and the broken curl example in runpod-flash. #923 rewrites the Flash quickstart and tutorials from the same live runs, and documents how 1.21.0 behaves until this ships. Review them together.

Calling a deployed @Endpoint function fails on cold starts even though the job is fine, and the curl example flash deploy prints can't run at all.

What was breaking

Reproduced against live endpoints with runpod-flash 1.21.0:

Cold start past 90s fails with a misleading error. /runsync holds a request for 90s by default, the same as the client timeout. When a worker takes longer (waiting for a GPU, downloading a model), the client gives up first: timed out after 90s. the endpoint may not be deployed. The text-generation tutorial hit this on its first run.
Raising FLASH_SENTINEL_TIMEOUT makes it worse. runsync returns IN_PROGRESS at 90s, and the sentinel returns that status dict as the function result. The SDXL and text-generation tutorials then failed with KeyError: 'image_base64' and KeyError: 'generated_text'.
The printed curl example fails. It sends {"input": {}}. Runpod drops a job with an empty input, so it never reaches the worker and fails after about 40s with job timed out after 1 retries.

Change

Send ?wait= on runsync, 10s below the client timeout (ai-api accepts 1-300s). Timeouts too short for that submit with /run.
Poll /flash/status/{id} with backoff until the job finishes. Raise on CANCELLED/TIMED_OUT, and on a pending response without a job id.
Retry network errors and 429/5xx while polling, raising on the 5th in a row. A non-JSON /status response also raises a RuntimeError naming the job. Any other 4xx from /status raises a RuntimeError naming the job and endpoint. Cancel the job and raise if it stays IN_QUEUE for 15 minutes (counted from submission). Polling keeps working on jobs that run for hours. A running job is bounded only by the endpoint's execution_timeout_ms, which defaults to 0 (no limit). FLASH_SENTINEL_TIMEOUT is now a per-request timeout, not a cap on the whole call. The README says so.
The curl example reads the function's required parameters from source, e.g. {"input": {"prompt": "<prompt>"}}. It uses {"__empty": true} for functions with none, which the handler already strips. For class endpoints it skips the method's receiver and adds "method" when the class has more than one.

Verification

Live, against deployed endpoints (quickstart matrix multiply, SDXL, GPT-2):

Scenario	1.21.0	This branch
SDXL cold start, `FLASH_SENTINEL_TIMEOUT=15`	`RuntimeError: timed out after 15.0s ... may not be deployed`	job pending after 5s, polled, image returned in 38s
SDXL cold start, `FLASH_SENTINEL_TIMEOUT=30`	passed (job finished just inside 30s)	job pending after 20s, polled, image returned in 34s
GPT-2 and SDXL cold start, `FLASH_SENTINEL_TIMEOUT=300`	`KeyError` on the output at 90.9s	covered by the polling path above
Quickstart, `FLASH_SENTINEL_TIMEOUT=11` and `=5`	n/a	jobs pending, polled, all three results correct (`=5` went through `/run`)
Printed curl example	`{"input": {}}`, job failed after 40s	`{"input": {"size": "<size>"}}`; with a real value it returned `COMPLETED`
Job running 110s, default timeout (90s)	fails at 90s	runsync returned `IN_PROGRESS` at 80.5s, polled, result at 127.5s
`POST /flash/cancel/{id}` through the sentinel	n/a	job moved to `CANCELLED`

One branch run failed with job timed out after 1 retries because I redeployed while an abandoned job was running and the old worker was replaced mid-job. A clean retry passed.

Unit tests: 2687 passed (-n auto -m "not serial") + 43 serial, coverage 86%. ruff check, ruff format --check, and mypy on flash_sentinel.py are clean. deploy.py has 3 mypy errors that are also on main. Each review fix has a test that fails on the code before it.

Review: five Codex passes on the diff, plus a fresh pass on the full PR that found a float overflow in the poll backoff after about 1,000 polls and a receiver-naming bug, both fixed. Two findings were deliberately left as they are:

Positional-only parameters: the generated handler calls f(**input) on main, so the example can't make them work.
Functions defined inside a module-level if block: the example falls back to {"__empty": true}.

Related Flash PRs

runpod-flash stops getting fixes on 2026-10-29, so the runpod/flash changes need to merge and release before then.

PR	What it does	Status (2026-10-27)	Requires before merge
runpod-workers/flash#113	Installs `torchvision` in the GPU load-balancer image	Merged, not released. Ships with release PR runpod-workers/flash#112 (1.6.1).	Done.
#390	Keeps torch's dependency tree out of `flash build` archives	Approved	runpod-workers/flash#112 merged so the 1.6.1 worker image is released.
#394	Waits for cold-start jobs instead of failing at 90s, and prints a working curl example after `flash deploy`	Approved, review round 1 addressed	Nothing.
runpod/docs#923	Matches the Flash quickstart and tutorials to a live run on 1.21.0	Approved, review round 1 addressed	Nothing. Documents 1.21.0 as released. Revisit the 90-second guidance once #394 ships.
#369	Reads runpodctl's top-level `apikey` as a credentials fallback	Draft	Nothing.

runsync hands back IN_QUEUE or IN_PROGRESS when a job outlives the server-side wait, which is common on cold starts and model downloads. The sentinel returned that status dict as the function result, so callers failed later with a KeyError. Poll the sentinel /status route until the job reaches a terminal state, and raise on CANCELLED and TIMED_OUT instead of returning the status dict.

runsync holds a request for 90s by default, the same as the sentinel's default client timeout, so on a cold start the client usually timed out before runsync could hand back IN_QUEUE/IN_PROGRESS. the /status polling never ran and callers saw a misleading 'endpoint may not be deployed' timeout. send wait=<timeout - 10s> so runsync returns a pending status the client can poll.

the example sent {"input": {}}. runpod drops a job whose input is an empty dict, so the job never reached the worker and failed after about 40s with 'job timed out after 1 retries'. read the function's required parameters from source and use them as placeholders, or the __empty marker the handler already strips when the function takes none.

- raise when runsync returns a pending status without a job id - retry transient network errors while polling /status (5 in a row max) - cancel and raise when a job stays IN_QUEUE for 15 minutes, instead of polling forever when no worker can start - curl example: use a real method name for multi-method class endpoints, omit 'method' for single-method classes, read only top-level definitions

- treat 429 and 5xx from /status like network errors (bounded retries) - say so when cancelling a stuck job fails instead of claiming it was cancelled - curl example: drop self/cls only as a class method's receiver

…unsync runsync waits at least 1s server-side, so a client timeout at or below the 10s margin plus that minimum could expire before runsync returned a job id. submit with /run in that case and poll.

…inition - start the 15-minute queue deadline when the job is submitted, so time spent waiting on runsync counts toward it - curl example: use the last definition of a redefined function, as python does

…rl example

…iver - long jobs overflowed float in get_backoff_delay after ~1024 polls (about 85 minutes); keep the attempt count at the cap - curl example: drop an instance method's first parameter whatever it's named, and keep it for static methods

…example

…ling

OneAsterisk

Claude review, round 1

Verdict: Approved. The cold-start polling and the curl example fix the reported failures, and the tests cover the error paths. One Medium is worth a follow-up.
Approval gate: 0 Critical, 0 High, 2 Medium+Low → approved.

Existing feedback

No prior reviews, inline comments, or conversation comments on this PR. All CI checks pass (Build Package is skipped by workflow config).

PR description vs code

Claim	Status	Where
`?wait=` on runsync, 10s below client timeout, clamped to 1-300s	✅	`flash_sentinel.py:78-82`, `:125-127`
Timeouts too short submit with `/run`	✅	`flash_sentinel.py:125-130`
Poll `/status` with backoff; raise on `CANCELLED`/`TIMED_OUT` and on pending without id	✅	`:198-231`, `:273-276`, `:164-169`
Retry network errors and 429/5xx "up to 5 in a row"	⚠️	`:213-214` raises on the 5th consecutive error, so 4 are tolerated (nit below)
Other 4xx from `/status` raises a `RuntimeError` naming job and endpoint	✅	`:222-227`
Cancel and raise after 15 min `IN_QUEUE`, counted from submission	✅	`:134`, `:195`, `:232-246`
"Running jobs are bounded by the endpoint's execution timeout"	⚠️	Flash defaults `execution_timeout_ms` to `0` (no limit). See Medium.
Curl example reads required params, `__empty` fallback, skips receiver, adds `method` only for multi-method classes	✅	`deploy.py:123-206`; matches `handler_generator.py:172-200`
Unit tests, ruff, mypy pass	✅	CI green on all 5 Python versions
Stack table lists this PR as "Draft"	⚠️	PR is not a draft at `715d876`. Stale status.

Findings

Medium

Polling a running job has no client-side bound under Flash defaults — src/runpod_flash/flash_sentinel.py:45-47. The comment, docstring (:186-188), and PR body say the endpoint's execution timeout bounds a running job. Endpoint(execution_timeout_ms=0) is the default (endpoint.py:440, serverless.py:283), and Runpod treats 0 as no limit. A hung handler now keeps await my_func(...) blocked until the 24h job TTL expires and /status returns 404. Before this PR, FLASH_SENTINEL_TIMEOUT capped the whole call. Now it only caps each HTTP request, and nothing lets the caller cap total wait. cli/docs/README.md:284 still calls it the "sentinel request timeout". Fix: either add a total-wait cap (env var or an IN_PROGRESS deadline), or correct the comments/body and update the README line to say it is a per-request timeout.

Low

Non-JSON 200 from /status escapes as a bare JSONDecodeError — flash_sentinel.py:228. Every other poll failure raises a RuntimeError naming the job and endpoint. A proxy HTML page on 200 loses that context. Wrap response.json() the same way as the 4xx branch.

Nits

flash_sentinel.py:49-50 The comment says 5 consecutive errors are "tolerated", but >= at :214 raises on the 5th. Say "raise after 5 consecutive", or use >.
flash_sentinel.py:207 The 8 cap is coupled to base=1.0 and _STATUS_POLL_MAX_SECONDS=5.0. A named constant or math.ceil(math.log2(...)) keeps them in sync if either changes.

Simplification opportunities

None worth a change. The diff is tight. _required_params (~50 lines) is the right size for what it handles (last-definition, receiver, qualified staticmethod).

Verified clean

get_backoff_delay clamps before jitter, so the exponent cap at :207 prevents the overflow and keeps the delay at ~5s ±20%.
httpx.TimeoutException subclasses TransportError, so a read timeout while polling is retried, not raised.
queued_since resets on IN_PROGRESS, so a worker that dies and requeues the job gets a fresh 15-minute window.
Manifest file_path is relative to the project root, and flash deploy uses Path.cwd() as the project dir (cli/utils/app.py:12), so _required_params resolves the right file.
functions[0] and class_methods[0] match what handler_generator.py:395 and :436 deploy.
rich.markup.escape covers the JSON body; the shell single-quote wrapper is safe because param names are Python identifiers.
sentinel_lb_request is unchanged; only QB paths poll.
Tests patch asyncio.sleep and time.monotonic, so none of them wait on real time.

Questions for the author

Is a fixed 15-minute queue cap right for scarce GPU types (H100/B200 scale-from-zero)? Before, a user could set FLASH_SENTINEL_TIMEOUT=3600 to wait longer in queue. Now the job is cancelled at 15 minutes regardless. Should it be configurable?
On KeyboardInterrupt or task cancellation during polling, the job keeps running server-side. That matches 1.21.0 runsync behavior. Was cancelling it on the way out considered?

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 30 in / 7,712 out / 902,662 cache read / 74,129 cache write = 984,533. Estimated cost: $2.09 at the rate card in the skill's sum_usage.py (cache writes priced as 5-minute, 1.25× input). Round 1 of this PR. This is automated assistance; a human owns the final call.}

…SON status, tie backoff cap to max delay

OneAsterisk

Claude review, round 2

Verdict: Approved. All three round 1 findings are fixed at 9da9066, and the new delta introduces no issues.
Approval gate: 0 Critical, 0 High, 0 Medium+Low → approved.

Status of prior findings

#	Finding	Severity	Status	Evidence
R1-1	Polling a running job has no client-side bound under Flash defaults; docs called `FLASH_SENTINEL_TIMEOUT` a request timeout without saying it no longer caps the call	Medium	Fixed (documented)	`flash_sentinel.py:35-37`, `:55-59`, `:199-203`; `cli/docs/README.md:284-287` now say it is per-request and that `execution_timeout_ms=0` sets no limit. PR body matches.
R1-2	Non-JSON 200 from `/status` escapes as a bare `JSONDecodeError`	Low	Fixed	`flash_sentinel.py:242-248` wraps it in a `RuntimeError` naming job and endpoint; covered by `test_poll_non_json_response_raises`
R1-3	Comment said 5 consecutive errors are "tolerated" but `>=` raises on the 5th	Nit	Fixed	`flash_sentinel.py:61-62` now says "raises after this many"; PR body says "raising on the 5th in a row"
R1-4	Hard-coded backoff exponent cap `8` coupled to base and max delay	Nit	Fixed	`flash_sentinel.py:47-53` derives `_STATUS_POLL_MAX_ATTEMPT = ceil(log2(5.0/1.0)) = 3`
R1-5	PR body stack table listed this PR as "Draft"	Nit	Fixed	Table now reads "Approved"

Existing feedback

No human or bot comments besides round 1. My three round 1 threads are now fixed and resolved.

PR description vs code

Claim	Status	Where
Non-JSON `/status` response raises a `RuntimeError` naming the job	✅	`flash_sentinel.py:242-248`
Retry network errors and 429/5xx, raising on the 5th in a row	✅	`flash_sentinel.py:226-235`
A running job is bounded only by `execution_timeout_ms`, default 0 (no limit)	✅	`flash_sentinel.py:55-59`; `endpoint.py:440` default
`FLASH_SENTINEL_TIMEOUT` is a per-request timeout; README says so	✅	`cli/docs/README.md:284-287`
Unit tests, ruff, mypy pass	✅	CI green at `9da9066` on 3.10 to 3.14, Nix flake, Build Package

Claims unchanged since round 1 were re-checked against the delta and still hold.

Findings

None.

Verified clean

Capping the exponent at 3 instead of 8 does not change the delay sequence: get_backoff_delay computes 1.0 * 2**attempt then clamps to 5s, so polls still go 1s, 2s, 4s, then 5s (±20% jitter).
httpx.Response.json() raises json.JSONDecodeError, and a bad encoding raises UnicodeDecodeError. Both subclass ValueError, so except ValueError catches both.
response.text[:200] slices code points, so it cannot split a character, and httpx decodes with errors="replace".
The new async test runs under the class's autouse asyncio.sleep patch (test_flash_sentinel.py:193), so it does not wait on real time.
The README paragraph matches the code: 15-minute queue cap, unbounded running jobs, execution_timeout_ms as the only bound.

Questions for the author

Carried from round 1, still unanswered. Neither blocks merge.

Should the 15-minute queue cap be configurable for scarce GPU types that scale from zero?
On KeyboardInterrupt or task cancellation during polling, the job keeps running server-side. Was cancelling it on the way out considered?

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 12 in / 3,214 out / 267,596 cache read / 49,751 cache write = 320,573. Estimated cost: $1.22 at the rate card in the skill's sum_usage.py (cache writes priced as 5-minute, 1.25× input). Round 2 of this PR. This is automated assistance; a human owns the final call.}

…ription

OneAsterisk

Claude review, round 3

Verdict: Approved. The only change since round 2 (5bbffa0) corrects the --python-version help text, and the new text matches the code.
Approval gate: 0 Critical, 0 High, 2 Medium+Low → approved.

Status of prior findings

#	Finding	Severity	Status	Evidence
R1-1	Polling a running job has no client-side bound; docs did not say `FLASH_SENTINEL_TIMEOUT` no longer caps the call	Medium	Fixed (round 2)	Unchanged at `5bbffa0`
R1-2	Non-JSON 200 from `/status` escapes as bare `JSONDecodeError`	Low	Fixed (round 2)	Unchanged at `5bbffa0`
R1-3	"tolerated" comment vs `>=` check	Nit	Fixed (round 2)	Unchanged
R1-4	Hard-coded backoff exponent cap	Nit	Fixed (round 2)	Unchanged
R1-5	Stack table listed this PR as Draft	Nit	Fixed (round 2)	Unchanged

Existing feedback

No human or bot comments besides my earlier rounds. All three round 1 threads are resolved.

PR description vs code

Claim	Status	Where
Sentinel and curl-example claims	✅	Verified in rounds 1 and 2; `5bbffa0` does not touch `flash_sentinel.py` or the curl code
`5bbffa0` help-text change	⚠️ Not mentioned	PR body has no line about `--python-version` or `python_version`

Findings

Low

Undocumented, out-of-scope change — build.py:567, deploy.py:54, serverless.py:279. The PR is about cold-start polling and the curl example; the Python-version help fix is unrelated and the body does not mention it. Either add one line under "Change" or move it to its own PR so the squash message matches what ships.
CLI docs still carry the old text — src/runpod_flash/cli/docs/flash-build.md:31 and flash-deploy.md:91 still say "3.10, 3.11, or 3.12" and "Overrides per-resource python_version". The new help text says 3.13 is supported and that a conflict fails. Update both lines in this commit so help and docs agree. (Not in the diff, so no inline comment.)

Verified clean

3.13 is supported: SUPPORTED_PYTHON_VERSIONS, GPU_PYTHON_VERSIONS, and CPU_PYTHON_VERSIONS all include it (core/resources/constants.py:8-18).
"Fails if it conflicts with a declared python_version" matches _reconcile_python_version (build_utils/manifest.py:447-462), which raises ValueError when the override differs from any resource's declaration. The old "Overrides" wording was wrong.
"Defaults to 3.12 when no resource declares one" matches return DEFAULT_PYTHON_VERSION (manifest.py:478, constants.py:15). The old "local interpreter version" description was wrong.
deploy passes python_version into the same build path (deploy.py:85), so the shared wording applies to both commands.
No test asserts the old help strings.
CI at 5bbffa0 is green: Quality Gates 3.10 to 3.14, Nix flake, Build Package, Validation.

Questions for the author

Carried from round 1. Neither blocks merge.

Should the 15-minute queue cap be configurable for scarce GPU types that scale from zero?
On KeyboardInterrupt or task cancellation during polling, the job keeps running server-side. Was cancelling it on the way out considered?

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 26 in / 5,420 out / 641,850 cache read / 53,274 cache write = 700,570. Estimated cost: $1.50 at the rate card in the skill's sum_usage.py (cache writes priced as 5-minute, 1.25× input). Round 3 of this PR. This is automated assistance; a human owns the final call.}

OneAsterisk · 2026-10-27T18:36:54Z

-            "Target Python version for worker images (3.10, 3.11, or 3.12). "
-            "Overrides per-resource python_version declarations. "
-            "Defaults to the version declared on resource configs, or 3.12 if none set."
+            "Target Python version for worker images (3.10, 3.11, 3.12, or 3.13). "


Low: This help-text fix (and the matching ones in deploy.py and serverless.py) is unrelated to cold-start polling and the curl example, and the PR body does not mention it. Add one line under "Change", or split it into its own PR. While here, cli/docs/flash-build.md:31 and flash-deploy.md:91 still list only 3.10 to 3.12 and say "Overrides per-resource python_version"; update them to match.

justinwlin added 8 commits October 27, 2026 10:13

fix(sentinel): retry transient status errors and report failed cancels

5d467d4

- treat 429 and 5xx from /status like network errors (bounded retries) - say so when cancelling a stuck job fails instead of claiming it was cancelled - curl example: drop self/cls only as a class method's receiver

fix(sentinel): submit with /run when the timeout leaves no room for r…

00577dc

…unsync runsync waits at least 1s server-side, so a client timeout at or below the 10s margin plus that minimum could expire before runsync returned a job id. submit with /run in that case and poll.

fix(sentinel): count queue time from submission and read the last def…

202baee

…inition - start the 15-minute queue deadline when the job is submitted, so time spent waiting on runsync counts toward it - curl example: use the last definition of a redefined function, as python does

fix(deploy): read the last definition of a redefined class for the cu…

d6efc3e

…rl example

justinwlin changed the title ~~fix(sentinel): wait for cold-start jobs instead of returning their status~~ fix: wait for cold-start jobs and print a runnable curl example after deploy Oct 27, 2026

This was referenced Oct 27, 2026

fix(build): keep torch's dependency tree out of tarballs #390

Open

fix(credentials): read runpodctl top-level apikey as fallback #369

Draft

docs(flash): match the quickstart and tutorials to a live run on runpod-flash 1.21.0 runpod/docs#923

Open

justinwlin added 3 commits October 27, 2026 12:07

fix(deploy): recognize qualified staticmethod decorators in the curl …

14d6982

…example

fix(sentinel): raise a RuntimeError naming the job on a 4xx while pol…

715d876

…ling

justinwlin marked this pull request as ready for review October 27, 2026 17:42

OneAsterisk approved these changes Oct 27, 2026

View reviewed changes

Comment thread src/runpod_flash/flash_sentinel.py Outdated

Comment thread src/runpod_flash/flash_sentinel.py Outdated

Comment thread src/runpod_flash/flash_sentinel.py Outdated

fix(sentinel): address review: document unbounded polling, wrap non-J…

9da9066

…SON status, tie backoff cap to max delay

OneAsterisk approved these changes Oct 27, 2026

View reviewed changes

fix(cli): correct --python-version help and python_version field desc…

5bbffa0

…ription

justinwlin mentioned this pull request Oct 27, 2026

fix(docker): install torchvision in GPU load-balancer image runpod-workers/flash#113

Merged

OneAsterisk approved these changes Oct 27, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix: wait for cold-start jobs and print a runnable curl example after deploy - #394

fix: wait for cold-start jobs and print a runnable curl example after deploy#394
justinwlin wants to merge 13 commits into
mainfrom
fix/sentinel-poll-in-progress

justinwlin commented Oct 27, 2026 •

edited

Loading

Uh oh!

OneAsterisk left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OneAsterisk left a comment

Uh oh!

OneAsterisk left a comment

Uh oh!

OneAsterisk Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

justinwlin commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was breaking

Change

Related Flash PRs

Uh oh!

OneAsterisk left a comment

Choose a reason for hiding this comment

Claude review, round 1

Existing feedback

PR description vs code

Findings

Medium

Low

Nits

Simplification opportunities

Verified clean

Questions for the author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OneAsterisk left a comment

Choose a reason for hiding this comment

Claude review, round 2

Status of prior findings

Existing feedback

PR description vs code

Findings

Verified clean

Questions for the author

Uh oh!

OneAsterisk left a comment

Choose a reason for hiding this comment

Claude review, round 3

Status of prior findings

Existing feedback

PR description vs code

Findings

Low

Verified clean

Questions for the author

Uh oh!

OneAsterisk Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

justinwlin commented Oct 27, 2026 •

edited

Loading