[planetf1]

What changes for users

A deployer can measure how many decision requests per second their server handles, and at what latency, with one command and no dependencies. The script takes the server's URL; nothing in it points at a particular endpoint.

Mechanism

scripts/load_test.py runs N clients, each on one keep-alive connection, sending a three-question request (predicate, 20-way choice, 5-level score) over a state of about 230 tokens. --mode unique gives each request a fresh state; --mode identical repeats one body, which the prefix cache serves almost entirely and so overstates throughput. It prints requests per second, p50/p95/p99 and the HTTP statuses per level, and the prefix-cache hit rate from vLLM's /metrics when present. docs/TESTING.md explains how to run it and how to read the results, including the effect of SYSTEMONE_MAX_CONCURRENT_REQUESTS (503s, not queueing) and --backoff.

Limitations

• One run per level and 45 seconds by default, so there is no measure of spread.
• The client is Python threads in one process; at a few hundred clients it may itself limit throughput.
• The prefix-cache counters are server-wide and count all traffic.
• It overlaps with the optional --load of probe_check.py (identical body, three levels, with chat alongside); this one adds unique content, more levels and the cache hit rate.

Testing

• ruff, ty check and the existing suite pass.
• Run against a local stub server: unique and identical modes, a server returning 503 for half the requests with and without --backoff, the prefix-cache line, and the error messages for a bad URL, bad levels, an unreachable host and a missing argument.
• Used against vLLM 0.30.0 with the 8b on one H100 from a separate node; those runs are not recorded here.
• Run from a separate node against the plugin on vLLM 0.30.0 (the 8b, one H100) with the final script: unique mode at 1 and 8 clients, identical mode at 8; all HTTP 200, and the prefix-cache hit rate line printed (low for unique, near 97% for identical).