Job-take round trip costs ~500ms with a deep queue, capping saturated worker throughput #647

Problem

With jobs already queued, a job-take round trip costs a flat ~500ms, whether it asks for 2 jobs or 32. A worker has only one job-take open at a time, so this caps a saturated worker's throughput regardless of GPU speed.

Even after #646, half of every take/run cycle on a fast vLLM worker is spent waiting on job-take:

• take 32 (~0.5s)
• run them (~0.5s)
• repeat

That's ~32 jobs/s, where the GPU could do roughly twice that.

Evidence

runpod/worker-v1-vllm:v2.28.0, Qwen2.5-0.5B-Instruct, RTX 4090, MAX_CONCURRENCY=32, one warm worker, bursts of 512 /runsync requests.

The worker ran main (aa72199) with an instrumented JobScaler.get_jobs (not merged). For every job-take it logged:

• the jobs_fetcher round trip
• jobs requested vs returned
• worker occupancy when the take was sent
• the worst event-loop lag during the take, from a 50ms timer

One 512 burst, showing that size doesn't matter:

• Not the worker's event loop. Loop lag during takes was 0–2ms (p90), so the request is sent and its response read promptly.
• Long-polls don't pay it. A job submitted to an idle worker with a take already waiting is taken in ~50ms (delayTime of sequential requests). The ~500ms shows up when a take is sent while jobs are already queued.

Ask

1. Where do the ~500ms go server-side when the queue is deep? (Batch-take wait window? Queue scan or locking? Per-request overhead?) Could the job-take service return immediately when it can fill the request?
2. If that can't come down, the SDK-side alternative is take-ahead: send the next take ~0.5s before slots free up, so it overlaps running jobs. That hides the latency, but a busy worker would then hold jobs an idle worker could start sooner. Is that acceptable for job distribution across workers, and should it be opt-in, e.g. only for high-concurrency workers?

Context

• perf(serverless): take a full batch once drained #646: full worker takes a full batch as soon as it drains instead of sleeping 1s (−4.6s, 19%, for a 512 burst). It works around this latency and doesn't remove it.
• Prototypes that took each freed slot right away were slower, because every take pays the same ~500ms. So the job-take round trip, not the SDK's batching, is now the throughput limit.