[deanq]

Summary

When a worker is at its concurrency limit, JobScaler.get_jobs slept a fixed 1s before checking for room again. With jobs queued, a job-take costs ~0.5s on the platform side, whatever its size. So a saturated 32-slot vLLM worker ran this cycle:

1. take 32 (~0.5s)
2. run them (~0.4s), inside the 1s sleep
3. sit idle for the rest of the sleep

That is one 32-job batch every ~1.4s, with the GPU idle for most of each cycle.

• A full worker now waits until no job is in progress, bounded by FULL_WAIT_SECONDS = 1 (the old sleep), then takes a full batch.
• handle_job sets an asyncio.Event when a job leaves the worker; the wait also ends on shutdown.
• The event is cleared before each occupancy re-check, so a job finishing in between re-arms it and is not missed.

Because the wait is bounded by the old sleep, a full worker takes the same or a bigger batch, no later than before. One long-running job can't hold the other slots idle for more than 1s, and set_scale() still runs at least as often.

Why not take each slot as soon as it frees: I prototyped that too, with and without a linger to batch. It was slower (512-job burst finished 99% of jobs 4–7s later). Every take pays the same ~0.5s, so smaller takes mean fewer jobs per round trip.

Impact

runpod/worker-v1-vllm:v2.28.0, Qwen2.5-0.5B-Instruct, RTX 4090, MAX_CONCURRENCY=32. One warm worker per arm, both arms loaded at the same time, 4 reps. The SDK is swapped via start command; the worker logs the patched file's sha256 to confirm which code ran.

burst of 512	main	this PR	Δ (95% bootstrap CI)
/runsync, time to 99% done	24.3s	19.7s	−4.6s [−5.9, −3.0]
OpenAI route, time to 99% done	24.4s	19.8s	−4.6s [−5.8, −3.3]
median delayTime	11.2s	8.7s	−2.4s
median time between job-takes	1.31s	0.85s

Sequential requests and 32-job bursts (worker never saturated) are unchanged. Single-slot workers also take as soon as their job finishes, since occupancy 0 is all slots free.

The remaining limit is the ~0.5s job-take round trip itself, which is half of every cycle now. See #647.

Test plan

• New: full worker takes a full batch as soon as it drains (fails on main)
• New: full worker does not take while only partly drained
• New: a straggler can't hold freed slots past FULL_WAIT_SECONDS
• New: full worker stops taking promptly on shutdown (fails on main)
• test_rp_scale.py repeated 5x, no flakes
• make quality-check: 1540 passed, 91.91% coverage

[Copilot]

🟡 Changes recommended

An empty worker with zero concurrency now enters a tight loop that starves other event-loop tasks.

1 open issue

• Avoid busy loop when concurrency modifier is zero · New

What changed in this PR

This PR improves saturated serverless worker throughput by replacing fixed polling delays with a bounded wait for the worker to drain, allowing earlier full-batch job acquisition.

Changes:

• Wake job acquisition when all slots are free, on shutdown, or after one second.
• Signal job completion through an asynchronous event.
• Add regression tests for full draining, partial draining, timeout, and shutdown.

File	Description
tests/​test_serverless/​test_rp_scale.py	Adds coverage for bounded batch waits and shutdown behavior.
runpod/​serverless/​modules/​rp_scale.py	Implements event-driven draining before fetching another batch.

_{Review effort: Balanced}