Repository navigation
Spur follow up work - #299
Spur follow up work #299Cemberk wants to merge 186 commits into
Conversation
… selector
The xPyD harness already models pools that span multiple nodes (xP/yD count
NODES, roles are MASTER/CHILD via --data-parallel-start-rank + --headless), but
the wideEP path hardcoded TP=1: dp_size was xP*GPUS_PER_NODE and the connector
emitted a literal `-tp 1`. That makes any model whose REPLICATED (non-expert)
weights exceed one GPU inexpressible - e.g. Kimi-K3 on MI300X, where TP1/DP16
needs 190.7 GiB/GPU (> 192 GB HBM) and TP2 is required to fit.
- vllm_disagg.sh: topology math is TP-aware. dp_per_node = GPUS_PER_NODE/TP_SIZE
feeds dp_size, dp_size_local and both start-rank calculations. TP_SIZE defaults
to 1 and is validated to divide GPUS_PER_NODE.
- moriio.sh: emit -tp ${TP_SIZE}; clamp --api-server-count to dp_size (it must be
<= data-parallel-size or the frontend DP balancer routes to ranks that do not
exist - only reachable once TP>1 shrinks dp_size below GPUS_PER_NODE).
- moriio.sh: advertise the peer pool's node IPs as moriio_pod_hosts when a pool
spans >1 node. Without it the connector falls back to the peer MASTER only and
KV writes aimed at ranks on a peer CHILD node silently miss, so those ranks
decode with no context. Emitted only when xP>1 || yD>1.
- run_xPyD_models.slurm: forward TP_SIZE/GPUS_PER_NODE, wire BENCHMARK_SCRIPT=niah
to the already-present benchmark_niah.sh, forward NIAH_* knobs.
Default-neutral: at xP=1 yD=1 TP_SIZE=1 - the topology every registered entry
uses - DRY_RUN output is byte-identical to develop across all four node roles.
At xP=2 yD=2 TP_SIZE=2 it emits tp=2, dp_size=8, dp_local=4, child start-rank 4
(= EP16 per pool over 16 GPUs).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…Kimi-K3 recipe
Two parts.
1) moriio.sh wideEP dropped ${model_args[@]} on the floor. models.yaml documents
per-model dp: tuning as supported ("both connectors now append it") and rixl's
deepep path does append it, but the moriio wideEP branch parsed the flags,
logged them, and then emitted an argv without them. Verified with a sentinel
flag: present in the log line, absent from the command. Now appended, matching
rixl. Every existing wideEP model has an empty dp: block, so emitted argv is
unchanged for all of them.
2) Kimi-K3 recipe, entirely as data: env: block for the gfx942 knobs
(VLLM_ROCM_USE_AITER_MLA=0 - the AITER MLA kernel is gfx950-only -
AITER_SITUV2_A8W4=1 for the packed-int4 SiTUv2 MoE path, KDA conv state layout,
40e9 KV cache for >600K contexts, per-role cudagraph + MoRI backends), and a
dp: block for the serve flags the connector does not emit (reasoning parser,
1M max-model-len, batched-tokens pinned at 2048, MoE quantization-config).
Registered in MORI_EP_VALID_MODELS and WIDE_EP_ONLY_MODELS.
The JSON quantization-config carries its own shell quotes: the launcher
word-splits dp: via eval, which would otherwise strip the JSON's double quotes
and hand vLLM invalid JSON. Verified the emitted value parses as JSON.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker/pyt_vllm_kimi_k3_mi300x.ubuntu.amd.Dockerfile
The shared vllm_disagg_inference stack with the pins K3 needs on gfx942
(MoRI 1.2.2, AITER 0.1.19 + flydsl 0.2.4, the K3+MoRIIO vLLM commit) plus the
K3-aware AITER graft from the public vendor image - without that graft the K3
MoE profiling shape finds no tuned FlyDSL config, falls back to a heuristic
kernel and aborts LLVM inside determine_available_memory.
Differences from the recipe this is ported from:
- every source pinned to an immutable SHA, not a fork branch name (branches on
personal forks can be force-pushed; MAD needs the image rebuildable later)
- GH_TOKEN build-arg dropped: all three repos are public, and a token passed
this way is recorded in image metadata
- the flydsl 0.2.4 re-pin happens at build time, so the launcher no longer runs
pip install inside every container at serve time
- WITH_NIXL defaults to 0 (K3 uses the moriio connector only)
- no runtime patchers: all connector/KDA fixes are committed in the pinned vLLM
scripts/vllm_multinode/
Colocated (single-instance) multi-node serving - the counterpart to
vllm_dissag, for models that do not fit one node but want lowest single-request
latency rather than disagg throughput. TP within a node, PP across nodes.
It owns only node discovery, the container launch and the head/worker split;
models.yaml, socket_barrier.py, benchmark_xPyD.sh, benchmark_niah.sh and
parse_to_csv.py are reused from vllm_dissag (scripts/ is mounted whole), so the
model recipe and the CSV pipeline have one home.
The three K3 colocated variants collapse to this one script: dry-run confirms
pp2xtp8 / allgather / moriep argv differ only by --enable-expert-parallel and
--all2all-backend, so they become three models.json entries rather than three
near-identical run.sh copies.
Not built here - this environment has no docker and one 8-GPU node; the harness
is verified by DRY_RUN argv inspection only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
models.json: four entries, all skip_gpu_arch gfx950 (the existing four Kimi-K3 entries are gfx950/TP8 single-node and all carry skip_gpu_arch gfx942, so K3 is currently skipped entirely on MI300X - this is that gap): pyt_vllm_disagg_mori_kimi-k3 -N 4 xP2 yD2 TP2 -> EP16/pool pyt_vllm_kimi-k3_mi300x_pp2xtp8 -N 2 TP8 x PP2, no EP pyt_vllm_kimi-k3_mi300x_wideep_allgather -N 2 + EP, allgather_reducescatter pyt_vllm_kimi-k3_mi300x_wideep_moriep -N 2 + EP, mori_low_latency The three colocated entries differ only in ENABLE_EP / ALL2ALL_BACKEND / AITER_SITUV2_A8W4, which is why they share one launcher. Result reporting: BENCHMARK_SCRIPT=niah previously produced logs and nothing else, so a model declaring multiple_results would have reported no metric. - parse_to_csv.py gains --niah: parses benchmark_niah.py output and writes the same 29-column madengine perf.csv as the throughput path (verified identical column sets). One row per context size, performance = needles found /10. A size that errored is written as a FAILURE row with performance 0 rather than dropped, so a pass->crash regression is visible instead of silent. - benchmark_niah.sh calls it. - Both slurm launchers copy /run_logs/$SLURM_JOB_ID/perf.csv to ./perf_$MODEL_NAME.csv on completion: madengine resolves multiple_results against the job directory it launched from, not the container log mount. No-op for entries that do not declare multiple_results. Verified: madengine discover goes 133 -> 137 with exactly these four added and none removed; every referenced dockerfile and script exists; -N matches each topology; the int4 quantization-config survives shell word-splitting as valid JSON; DeepSeek moriio and rixl/deepep DRY_RUN output is byte-identical to develop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ports the writeups from PR ROCm#193 next to the existing gfx950 K3 doc, keyed to the madengine tags rather than to shell scripts: topology rationale (why MI300X needs multi-node and why the disagg pool needs TP2), the gfx942 knobs and where each one lives now, the NIAH results, and the three root-cause fixes. Corrections carried in: - the NIAH ceiling is 900K throughout. The upstream READMEs still said 300K in seven places after two later commits raised it to 500K and then 900K. - states that the NIAH harness sizes context in WORDS (~1.3x tokens) while the results table is in tokens, so the two columns are not comparable. benchmark/kimi_k3/README.md gains a pointer: all four entries there carry skip_gpu_arch gfx942, which is exactly the gap the MI300X page fills. Every tag, node count and knob value in the new page is checked against models.json and models.yaml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…metadata The four MI300X Kimi-K3 entries could not run correctly on a cluster. Three separate defects, all from the colocated launcher inheriting conventions that only hold for the disaggregated one. Node sizing. The entries carried `"args": "-N 4 -n 4"` on the assumption that madengine forwards args to sbatch. It does not — args are appended to `bash <model>.slurm`, and neither launcher parses $@, so the flags were dropped and the job was submitted with the `slurm.nodes` default of 1. Sizing now uses `distributed.nnodes`, which madengine reconciles into `#SBATCH --nodes`, and the inert args are removed. `slurm.nodes` is deliberately left unset: setting it selects the multi-node preset, whose `NCCL_SOCKET_IFNAME=eth0` is forwarded into the container by run_xPyD_models.slurm and would override the fabric interface on an RDMA cluster. `slurm.time` is set explicitly instead, since the single-node preset's 12 h default is short for a full NIAH sweep. NIAH model tag. benchmark_niah.sh requested MODEL_PATH as the model name, correct for the disagg path where vLLM defaults served_model_name to the serve argument. serve_colocated.sh passes `--served-model-name "$MODEL_NAME"`, so every request 404'd and all three colocated entries — which default to BENCHMARK_SCRIPT=niah — would have recorded a full sweep of FAILURE rows. The harness now honors NIAH_MODEL when a launcher sets it, and the colocated launcher sets it to the tag it actually serves. Colocated run metadata. parse_to_csv derived nnodes, n_gpus, deployment_type and tags from xP/yD. serve_colocated.sh exports xP=1 yD=0 only so the shared log filenames stay unique, so a 2-node 16-GPU colocated run reported itself as a 1-node 8-GPU `disagg_1P0D` with a nixl backend it never used. Node count now comes from NNODES (exported by both launchers), and a launcher whose shape is not "xP prefill + yD decode" states its own identity via PERF_DEPLOYMENT_TYPE / PERF_TAGS. The disagg path is byte-identical to before. Also corrects the README, which documented the args-to-sbatch behavior that does not exist and a quick start that cannot work now that the "<supply-your-image>" placeholder is rejected rather than pulled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NUM_NODES is derived from the requested topology (xP+yD), not from the allocation, and the node list was then truncated with `head -n $NUM_NODES`. A short allocation therefore produced a silently wrong topology: pools were built from nodes that were never allocated, and the run failed much later as a connector handshake timeout with no indication of the real cause. This mattered little while these models were always launched from a matching `salloc`, but the sbatch path now sizes the allocation from the model card, so a mismatch between card and cluster is a reachable state. Check up front and name the three ways to fix it. The colocated launcher already fails on the equivalent mismatch via its TP*PP == NNODES*GPUS check, so it needs no counterpart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ntract parse_to_csv.py hand-wrote a full 29-column perf.csv, assembling node counts, GPU counts, launcher, image and tags from the environment. Most of that is not the workload's to know: madengine already owns it, and the guesses were wrong for the colocated launcher, which sets xP=1 yD=0 only to keep log filenames unique. The NIAH writer now emits a narrow CSV — model, performance, metric, status, plus descriptive columns — and madengine merges in the run metadata via the entries' `multiple_results` declaration. The columns that genuinely describe the workload's own configuration rather than its placement (tp, pp, ep_backend, prefill_decode) move to descriptive columns, matching how scripts/vllm/run_vllm.py already reports tp/dtype/bs on the templated path. Nothing is lost; it is relocated to whichever side actually knows it. `status` stays explicit: an errored context size scores 0, and deriving status from performance would file that real failure as a SUCCESS. Scoped to leave every other workload alone. The NIAH writer is reachable only by the four Kimi entries, which all declare `multiple_results`. The throughput writer keeps its full-schema output by default, because the twelve disagg cards that use it declare no `multiple_results` and madengine reads their CSV directly with no metadata to merge — a narrow CSV there would drop every descriptive column. Their output is byte-identical; `--narrow` is the opt-in for migrating one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uted.nnodes
madengine reads only slurm.nodes when emitting `#SBATCH --nodes`
(deployment/slurm.py: `self.nodes = slurm_config.get("nodes", 1)`), and
nothing maps distributed.nnodes across — build_orchestrator copies nnodes
into deployment_config.distributed for launcher detection only. The four
MI300X model cards carried nnodes but no slurm.nodes, so every one of them
would have been submitted as a 1-node job. The launcher's
TP*PP == NNODES*GPUS_PER_NODE assertion catches it, but only after the
allocation is granted.
Add nodes/gpus_per_node to the four slurm blocks, and ship the site configs
that carry the values a model card cannot:
- results_dir, absent from the key list madengine copies out of models.json,
and required because the slurm_multi collector globs results_dir for
perf*.csv rather than resolving multiple_results
- MODEL_DIR / LOG_PATH, which otherwise default to /shared_inference
*.json is gitignored repo-wide, so the templates need explicit negations or
they never reach a clone.
Correct the two README claims that did not match madengine's behavior
(nnodes-sizes-the-allocation, multiple_results-is-resolved-first), document
that --account/--qos are unwired for slurm_multi and need SBATCH_ACCOUNT,
and drop --keep-model-dir from the SLURM examples — it is a local-Docker
flag that madengine ignores with a warning on SLURM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four defects, each found by running the recipes end to end on a real
MI300X SLURM cluster for the first time. None of them could have survived
a single successful run, which is consistent with these cards never having
been executed through madengine.
1. Build was not reproducible. Every source is pinned to a commit SHA, but
pip resolves build dependencies in an isolated environment from PyPI at
build time, so nothing pinned the toolchain. setuptools >= 80 added
assert isinstance(self.compiler, CCompiler)
to distutils' build_ext.build_extension, which MoRI's legacy
Cython.Distutils.build_ext path violates, so the amd_mori wheel failed
with "AssertionError: run() must precede build_extension()" while every
pinned SHA was still correct (job 223849). PIP_CONSTRAINT is the only
mechanism that reaches inside pip's build isolation -- installing
setuptools in the image does not. Set globally so the AITER, vLLM and
router stages cannot regress the same way. Build then completed in
19 minutes, all 18 stages (job 223872).
2. RDMA library mounts used -e, which is true for directories. Docker
creates a DIRECTORY at a bind-mount source that does not exist, so the
first run on a node lacking e.g. libionic.so.1 leaves an empty directory
behind, and every later run on that node mounts a directory over a file
inside the image:
OCI runtime create failed: ... not a directory
Self-propagating and per-node, so it presents as intermittent. The
surviving node then waits at socket_barrier.py forever for a peer that
already died -- 4h10m of "Waiting for nodes. . ." before the wall clock
(job 223909). Use -f so only regular files are mounted.
3. pyt_vllm_kimi-k3_mi300x_pp2xtp8 omitted the gfx942 MoE requirement that
the two wideep variants carry. gfx942 has no scaled-MXFP4 MFMA and the
a16w4 SiTUv2 heuristic FlyDSL kernel cannot codegen there, so the MoE
must be requantized to packed int4. Without it the worker aborts inside
determine_available_memory with
LLVM ERROR: Do not know how to expand this operator's operand!
and quantization_config=None in the engine config (job 224132). This is
a hardware requirement, not an EP-specific tuning, so all three
colocated cards now carry AITER_SITUV2_A8W4=1 and the int4 config.
4. A run that produced no results exited 0. The launchers warned about a
missing perf CSV and returned success anyway, so madengine recorded
exit_code=0 in its completion marker and the collector reported
"0 successful, 0 failed" -- a hard engine crash was indistinguishable
from a clean run. For a benchmark repo that is the worst possible
reporting outcome. Both launchers now exit 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of these cost hours to diagnose on a real cluster, and in every case the symptom is a long way from the cause -- an intermittent multi-node hang that is actually a poisoned bind-mount path, a wheel build that fails while every pinned SHA is correct, an LLVM abort that is a missing quantization flag. Record them so the next person reads instead of rediscovers. Also record the cluster facts a model card structurally cannot carry: the SLURM association wall-time cap (which silently parks a job as PENDING rather than failing), account requirements, docker on compute nodes, and the fact that a registry-less cluster has no supported path to distribute an image -- --build-on-compute requires --registry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds Kimi-K3-MXFP4 (MI300X gfx942 MXFP4 MoE) to the vllm_dissag framework: - models.yaml: K3 recipe env anchor + model entry (TP2×DP8, MoRI-EP) - run_xPyD_models.slurm: allowlists + 2P/2D validation + JIT cache split - moriio.sh: K3-gated topology (pod_hosts, api-server-count, headless KV) - vllm_disagg.sh: pod hosts export + _model_config_to_array helper - argv_assert.sh: K3 offline tests Config-only integration; image build documented in README. Co-Authored-By: Claude <noreply@anthropic.com>
…lsifiable
Both found on the first end-to-end MI300X run (job 224200), which served
Kimi-K3 and produced results -- but results that could not be trusted, from
a job that then would not exit.
Deadlock on teardown. serve_colocated.sh has the head kill only its own
server and exit, while every worker sits in `wait` on a headless vLLM that
nothing stops. srun waits on those tasks, so the job holds all nodes until
the wall clock: 4.5 hours of two idle exclusive MI300X nodes AFTER the
benchmark had already written perf.csv. The head now raises a sentinel on
the shared log volume and workers watch for it. The sentinel is raised from
an EXIT trap so a head that fails or times out cannot strand workers either.
NIAH could not distinguish truncation from retrieval failure. Scoring reads
content + reasoning_content, and K3 is a reasoning model served with
--reasoning-parser kimi_k3, so a trace that exhausts max_tokens is cut off
and only the earliest needles survive. At the old 2048 default, two of four
context sizes scored 1/10 and both listed exactly ANIMALS[0] -- the needle
placed first in the haystack. That signature is a truncated trace, not a
model failure, but finish_reason was never recorded so the two were
indistinguishable, and every row was written SUCCESS regardless.
- record finish_reason and print it as finish=<reason>
- raise the default max_tokens to 8192
- keep the needle count as the metric, always reported, never dropped
- mark a truncated row FAILURE and annotate the metric, so a measurement
artifact cannot enter the results as if it were a model result
9/10 stays SUCCESS: the README documents ~9/10 at >=20K as a known RDMA
residual, so a strict 10/10 gate would flag known-acceptable behaviour.
Logs predating the finish= field still parse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 239755 (Kimi-K3, 2x8 on oci-64) died in vLLM's startup barrier. Node 0 finished loading weights at 23:21:41 and entered in_the_same_node_as -> torch.distributed.barrier; node 1 was still constructing MoE layers ~50 min later. gloo gave up at 23:51:41, exactly 1800s in, and the master's TCPStore died with it, so node 1 then failed with "Broken pipe" to :29500. The barrier is all-or-nothing across all 16 ranks, so what has to stay small is the GAP between nodes, not the absolute load time. Five problems, addressed here: 1. --distributed-timeout-seconds does not reach the barrier that failed. It feeds ParallelConfig.distributed_timeout_seconds, which only configures the device/NCCL groups. The startup barrier runs on the gloo CPU group, built by GroupCoordinator from the separate cpu_distributed_timeout_seconds field. Unset, it falls back to PyTorch's stock 1800s -- which is exactly the deadline in the traceback, despite 7200 being passed. Pass --cpu-distributed-timeout-seconds too. 2. Cold AITER JIT compile on every rank, every run. The image points AITER_JIT_DIR/TRITON_CACHE_DIR/VLLM_CACHE_ROOT/COMGR_CACHE_DIR at /opt/vllm_cache, but this launcher mounted only /tmp/vllm_cache, a path nothing reads, inside --rm containers. Mount a host-persistent cache at /opt/vllm_cache keyed by image ID, as vllm_dissag/run_xPyD_models.slurm already does. Per-node cold compile is precisely the skew the barrier cannot absorb. 3. WORKER_PID was tee's PID. After `a | b &` the shell reports b, so every kill in this script hit tee and left vLLM running, and no liveness check was possible. Launch through process substitution instead. 4. A dead engine was indistinguishable from a slow one. The head polled the log for the full 4000s after its writer had already died, then reported a timeout rather than the traceback. Both wait loops now check liveness -- via a helper that also rejects zombies, since an exited-but-unreaped child still answers kill -0. This also fixes a latent hang where a worker would spin forever on a dead engine. 5. Container env was a closed allowlist. Tuning any of the above from CI meant a new -e line and another MAD PR. COLOCATED_FORWARD_ENV names the variables to forward, and they are passed via docker --env-file so values containing spaces survive. The file is node-local, mode 600, and unlike the wrapper scripts is never archived as a build artifact. Also adds an opt-out checkpoint page-cache pre-warm before the container barrier (PREWARM_CHECKPOINT=0 to skip). Its per-node duration is the real diagnostic: a large spread means the storage path is the problem and no timeout will fix it. Verified: bash -n clean on both files, and on the single-quoted srun body extracted in isolation; env-file forwarding preserves spaces and skips unset vars at mode 600; the liveness helper correctly reports an unreaped child as dead; DRY_RUN=1 emits the new flag and honours an override, with COLOCATED_EXTRA_ARGS still appended last. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 240624 spent its entire 7200s allocation in the pre-warm added by the previous commit. Both nodes printed "[prewarm] reading ..." and neither ever printed "[prewarm] done", so vllm serve was never launched -- the run timed out having tested nothing. The flaw: it reads the WHOLE checkpoint on EVERY node, while PP2xTP8 means a node only ever loads its own shard. On a 1453 GiB checkpoint that is roughly 4x the necessary I/O against a single NFS export, with both nodes competing for it. It manufactured exactly the contention it was meant to relieve. Default it to off. It stays available, because the per-node duration is a useful storage probe -- a large spread between nodes means the storage path is the problem and no timeout will fix it -- but it is now opt-in via PREWARM_CHECKPOINT=1 and bounded by PREWARM_TIMEOUT_SECONDS (default 900). On expiry it warns that the filesystem could not deliver the checkpoint in the window and proceeds, rather than burning the whole allocation. Note this run did not exercise --cpu-distributed-timeout-seconds: vLLM never reached the startup barrier, so the absence of the 1800000ms gloo timeout in the log says nothing either way. That fix is still untested. Verified: default unset is silent; PREWARM_CHECKPOINT=1 runs and reports its duration; a stalled read hits the budget, warns, and exits 0 so startup continues. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mori_ep_env.sh:72-73 sets
export NCCL_DEBUG="${NCCL_DEBUG:-INFO}"
export NCCL_DEBUG_SUBSYS="${NCCL_DEBUG_SUBSYS:-INIT,NET,GRAPH}"
The ${VAR:-default} idiom advertises that a caller can override these. No
caller can. run_xPyD_models.slurm hands the container a fixed 22-name -e
list, and NCCL_DEBUG is not on it, so whatever the submitter exports is
dropped at the container boundary and the default always wins. The override
contract in mori_ep_env.sh is dead code as far as anything outside the
container is concerned.
Two things follow from that, and neither is hypothetical -- both come from
job 246618, a Qwen3-32B 1P1D run that completed successfully:
1. Log volume. That run archived 19 MB of per-rank logs (decode_NODE1.log
9.6M, prefill_NODE0.log 4.4M, proxy_NODE0.log 2.7M), overwhelmingly NCCL
channel-tuning triples -- pre-adjustment / minNChannels / post-adjustment
-- emitted by all 8 ranks for every collective size.
2. Corrupted lines. Those 8 ranks share one stdout with no line atomicity,
so at INFO volume the writes interleave mid-token:
useocp[2026-09-07 16:09:52 TP0] Prefill batch, ...
justment based on threadThreshold
80 nc:112
Any grep-based marker extraction over these logs is unreliable, which
matters because CI reads them to decide pass/fail.
Narrowing NCCL_DEBUG_SUBSYS is not the fix. This run already had the
narrowed value INIT,NET,GRAPH and still produced all of the above; the
channel-tuning lines are GRAPH. Only NCCL_DEBUG=WARN suppresses them.
Forward the three NCCL debug knobs conditionally, using the same
${VAR:+-e VAR=$VAR} idiom vllm_dissag/run_xPyD_models.slurm already uses for
AITER_JIT_DIR and VLLM_ROCM_USE_AITER_MLA. When the variable is unset the
expansion is empty and the docker run is byte-identical to before, so the
default INFO behaviour is unchanged for anyone who does not opt out. Add the
same treatment for any further knob by adding one line.
The submitter's environment reaches the compute node because sbatch defaults
to --export=ALL and srun propagates it, so an `export NCCL_DEBUG=WARN` before
sbatch is now enough.
Deliberately not changing the INFO default here. INFO is a poor default for a
benchmark harness that archives its logs, but that is a call for the people
who use this interactively; this commit only makes the documented override
work.
models.json forced USE_CX7_NICS=0 on every sglang_disagg entry, which sends prefill->decode KV transfer over mlx5_1, the 100G management NIC, instead of the 8x 400G CX7 rail NICs that mori_ep_env.sh defaults to. That is the critical path of a disaggregated run, so the measured throughput does not represent the hardware. Two runs on different node pairs (AMD-internal 247073 and 247580) both show --disaggregation-ib-device mlx5_1. Flipped to 1 on the four models this evidence covers: llama-3.1-8b, qwen3-32b, llama-3.3-70b-fp8 and llama-3.1-405b-fp8. The remaining twelve entries keep 0 until someone benchmarks them. The rail NICs require same-rail nodes, and the file header has been citing select_same_rail_nodes() as the thing that enforces that -- a function which was never written. Written now: it maps each allocated node to its Level=0 switch via scontrol show topology and aborts before any container starts if they do not share one. A cross-rail pair hangs in NCCL ibv_modify_qp rather than failing, and base_flags sets --watchdog-timeout 1000000, so without this the job burns its whole allocation instead of failing in seconds. The check is skipped entirely when USE_CX7_NICS != 1, degrades to a warning when no topology is configured, and can be bypassed with REQUIRE_SAME_RAIL=0 -- topology.conf may describe the management network rather than the CX7 fabric, in which case the check is conservative rather than correct. --switches=2 is deliberately unchanged: both observed allocations were already single-leaf, so constraining placement is the one change here that could move the numbers.
decode.tp asked for --cuda-graph-bs $(seq 1 512), so 512 CUDA graphs were captured at startup while benchmark_xPyD.sh only ever sweeps CON="8 16 32 64 128 256 512". Startup was 51-52% of wall clock across two runs (426s of 810s, then 401s of 784s). Set the list to those seven values on the three models that used the full range. Llama-3.1-405B keeps $(seq 1 8): it is already small and raising it has memory implications this evidence does not cover.
py-spy, flask and pyyaml were installed from PyPI on every node at the start of every run, so the pinned image was mutated before each benchmark: two runs of the same tag are not the same environment, resolution can pick a different version than the one requested, and a PyPI outage fails the run. Observed in AMD-internal 247073/247580, where pip resolved click 8.5.0 and installed 8.4.1, and the force-reinstall surfaced an unrelated xgrammar / apache-tvm-ffi conflict in the log on both nodes. Baked into docker/sglang_disagg_inference.ubuntu.amd.Dockerfile. The launcher scripts now install only what is genuinely missing, so a MAD-built image does no work there while an image supplied through DOCKER_IMAGE_NAME still self-heals. --ignore-installed --force-reinstall is dropped.
Brings the branch onto current develop (652bad5) so the sglang_disagg fixes and the kimi-k3 work can ship in one PR. Eight files conflicted: models.json develop's 61b5301 migrated the registry from the root file to per-directory models.json. Took develop's empty root and moved this branch's four entries to where they now belong: pyt_vllm_disagg_mori_kimi-k3 into scripts/vllm_dissag/models.json, and the three pyt_vllm_kimi-k3_mi300x_* entries into a new scripts/vllm_multinode/models.json. The 26 jax_maxtext and primus_train entries in the old root file were NOT migrated: develop generates those from scripts/{jax-maxtext,primus_train}/get_models_json.py. scripts/sglang_disagg/run_xPyD_models.slurm Union of develop's 25 conditional -e forwards and this branch's three NCCL_DEBUG forwards (28 total). The CX7 flip that was applied to the old root models.json is re-applied to scripts/sglang_disagg/models.json here. scripts/vllm_dissag/models.yaml Append/append. Kept both recipe blocks (develop's GLM-5.1-FP8, this branch's Kimi-K3). scripts/vllm_dissag/connectors/moriio.sh Took develop's argument order (model_args last) from a33b622/b33923b, so recipe flags win over connector defaults. Compatible with the Kimi-K3 recipe. scripts/vllm_dissag/benchmark_niah.py and .sh The two sides added orthogonal features: develop added multi-seed sweeps with mean/min/max plus a warmup pass, this branch added finish_reason so a truncated answer is distinguishable from a retrieval miss. Combined both: _request now returns (message, finish, error) and the cross-seed summary flags TRUNCATED. Kept develop's NIAH_MAXTOK=2048 default -- the truncation detector makes the smaller default safe, and NIAH_MAXTOK is -e forwarded for models that need more. scripts/vllm_dissag/parse_to_csv.py The auto-merge left two parse_niah_log and two save_niah_perf_csv definitions; the later pair (this branch's found= parser) silently shadowed develop's. Since the merged summary emits develop's mean=/min=/max= format, dropped this branch's pair and kept develop's parser. Kept the --narrow flag. scripts/vllm_dissag/run_xPyD_models.slurm Six hunks, all unions: MORI_EP_VALID_MODELS and WIDE_EP_ONLY_MODELS gain both GLM-5.1-FP8 and Kimi-K3; the BENCHMARK_SCRIPT case gains both agentic and niah; the -e list gains develop's NIAH_SEEDS/NIAH_WARMUP/SHAPE_WARMUP/ USE_INDUCTOR_GRAPH_PARTITION alongside this branch's NIAH_* forwards. Verified: no conflict markers; bash -n on all eight touched shell/slurm files; py_compile on both Python files; json.load on all 21 registries and yaml.safe_load on both models.yaml; benchmark_niah.py run against a stub server on both the scoring and truncation paths and its log fed through parse_to_csv.py --niah to a valid perf.csv; and the sglang rail guard replayed against the real job-247580 topology across all six cases.
…d step Loading Qwen3-32B off /shared_inference measured 683s in job 249926 -- 67% of a 1015s bring-up, ~92 MB/s per node with both nodes pulling the same 63 GB through the same NFS mount. scripts/common/preflight_weights.sh:2-5 records what the other path is: /mnt/m2m_nobackup/models_blog is per-node local NVMe, non-uniform, while /shared_inference is the uniform NFS share. An earlier commit on this branch reordered the sglang launcher to probe NFS first, to avoid a wasted srun round trip and the "<jobid>.N bash FAILED 1:0" row a missed probe leaves in sacct. That trade is backwards: it spends 683s of NFS reads to save a few seconds of probing, and it means a node that does have the weights locally can never use them. Restored local-NVMe-first. The sacct noise that motivated the reorder is fixed properly instead. check_model_path now runs a probe that always exits 0 and decides from its output, so a miss in a fallback chain is not recorded as a step failure. Two latent bugs fall out of the same rewrite -- the old version read srun's exit code alone, so an srun that exited 0 while reporting from fewer nodes than were allocated, or one that exited 0 with no output at all, both returned "available on ALL nodes" and sent the job on to load weights from a path some rank cannot see. The new version requires one Found line per allocated node. Applied to both scripts/sglang_disagg and scripts/vllm_dissag; only sglang needed the ordering change, vllm_dissag was already local-NVMe-first. Verified by extracting the function and replaying it against a stubbed srun across five cases (all-found, one-node-miss, srun-fails, short output, silent exit-0). New: 0,1,1,1,1. Old, same stub: 0,-,1,0,0 -- it accepts short and silent. Note this does not by itself make Qwen3-32B load fast: jobs 247073, 247580 and 249926 all fell through to NFS, so that model is not currently staged on the local NVMe of the nodes CI draws from. This makes the fast path reachable and cheap to probe; the staging itself is a cluster-side action.
The merge moved four entries out of the root models.json into the per-directory registries develop's 61b5301 introduced, but kept their root-relative paths. All twenty pre-existing per-directory registries resolve dockerfile as ../../docker/X and scripts as a bare filename, so the migrated entries pointed at scripts/vllm_multinode/scripts/vllm_multinode/run_multinode.slurm and would not have resolved at all. All 117 entries across the 21 registries now resolve from their own directory.
…t Kimi-K3 offline The merge brought develop's multi-seed NIAH support into benchmark_niah.sh, which now reads NIAH_SEEDS and NIAH_WARMUP alongside NIAH_WORDS/MAXTOK/TIMEOUT. scripts/vllm_dissag forwards all of them; scripts/vllm_multinode forwarded only three, so a colocated niah run silently fell back to the defaults for seeds and warmup no matter what was set at submit time. Also forwards SHAPE_WARMUP, which benchmark_xPyD.sh reads and which is reachable from the colocated path via BENCHMARK_SCRIPT=sweep. Both launchers now forward every NIAH_* the benchmark reads. Kimi-K3 is the model this branch adds and the only cell argv_assert.sh did not cover -- it tested moriio+TP on Llama and moriio+wideEP on DeepSeek-V3. Added a Kimi-K3 section over both roles: the int4_per_group_32 requant (without which gfx942 cannot codegen the a16w4 SiTUv2 kernel and vLLM dies with "LLVM ERROR: Do not know how to expand this operator's operand!" in determine_available_memory, per benchmark/kimi_k3/mi300x/README.md:160), exactly one --compilation-config per role, the kimi_k3 reasoning parser, the prefill/decode split across cudagraph_mode (NONE vs PIECEWISE), all2all backend (mori_high_throughput vs mori_low_latency) and kv_role, and the two model-gate arrays. The gate assertions are scoped to the arrays themselves rather than grepped over the whole file: "Kimi-K3" appears in both MORI_EP_VALID_MODELS and WIDE_EP_ONLY_MODELS, so a file-wide grep passed even with the model removed from one of them. MORI_EP_VALID_MODELS is line-continued, so it is extracted as a block. 49 passed, 0 failed. Each new assertion group was mutation-checked against the merged tree: removing --quantization-config from the Kimi-K3 recipe fails 3, swapping the decode all2all backend fails 2, dropping Kimi-K3 from either gate array fails 1 each. The first version of the gate assertion did not fail under its mutation and was rewritten.
benchmark_xPyD.sh looped over "{1..$BENCHMARK_ITR}". Brace expansion runs before
parameter expansion, so that is the literal 6-character string, and the loop body
executed exactly once no matter what the variable held. Job 252775 ran with
BENCHMARK_ITR=2, printed "iter: {1..2}" into its own benchmark log, and swept each
concurrency once. scripts/vllm_dissag/benchmark_xPyD.sh already used seq.
The fix uncovers a second bug that the first one had been hiding. parse_to_csv.py
starts at "RUNNING: the benchserving script for iter: 1" and pairs each result block
with the RUNNING line in the section before it -- but an `if i > 1` guard skipped the
very first pairing. That was invisible while the marker never appeared: the search
failed, parsing began at byte 0, and the skipped section was the pre-warmup preamble,
which carries no RUNNING line anyway. Restore the marker and the skipped section
becomes the first sweep cell. Replaying job 252775's log with the loop fixed and the
parser untouched gives 6 rows instead of 7, silently missing con=8.
So: seq for the loop, drop the guard, and accept an optional prompt count in the
RUNNING regex so archived logs in the older format still parse. Replayed against
job 252775's benchmark log, the fixed parser reproduces the CSV that build 44
published, byte for byte, from a log in either format.
Fixing the loop alone would also have doubled every sglang_disagg sweep, since
run_xPyD_models.slurm defaulted the knob to 2. That default was only ever decorative,
so it moves to 1: observed behaviour is unchanged and the knob now works.
These comments cite the job IDs and measurements that motivated each fix. On a long-lived branch those numbers are unresolvable -- a reader six months from now cannot look up job 252775 -- while the mechanism they illustrate is permanent. Keep the mechanism, drop the identifiers. Comment-only: no changed line is code.
run_multinode.slurm runs under `set -euo pipefail` and expands COLOCATED_FORWARD_ENV at line 153 and again inside the srun body. Nothing assigns it - not the pipeline, not any model card - and `export NAME` on an unset variable marks the name for export without giving it a value, so it stays unset. Under `set -u` that expansion is fatal. It is an optional knob, so unset is the normal case, and the launcher aborted on the normal case: SLURM job 354061 printed its banner and died with "COLOCATED_FORWARD_ENV: unbound variable" before touching a model. Every vllm_multinode run has been failing this way, which is every Kimi card. Introduced in db276a6, which added the forwarding mechanism and both use sites but no default. Reproduced under `set -euo pipefail`: unset -> exit 1 with that exact message; with the default -> exit 0 and an empty forward list, and a set value still forwards, single and comma-separated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
run_multinode.slurm hardcoded MODEL_DIR=/shared_inference/models_blog, so a colocated run always pulled its checkpoint over NFS -- even on nodes that already held a local NVMe copy. It is the only one of the three launchers that did this; sglang_disagg/run_xPyD_models.slurm and vllm_dissag/run_xPyD_models.slurm have both probed /mnt/m2m_nobackup first for a while. It also never bind-mounted /mnt/m2m_nobackup into the container, so pointing MODEL_PATH there by hand could not have worked either. This matters most for exactly the models people complain about. A cluster survey found Kimi-K3 staged on local NVMe on all ten nodes that answered, at full size, while every colocated run read it over NFS. Small models hide the difference because they fit in page cache; a 1.5 TB checkpoint cannot. The probe requires the path on EVERY allocated node -- ranks share one MODEL_PATH, so a partial hit is useless -- and falls back to NFS otherwise. An explicit MODEL_DIR or MODEL_PATH still wins. The srun body always exits 0 so a miss does not leave a FAILED step in sacct, and the mount is guarded on -d because docker auto-creates a directory at a missing bind source, which would shadow the model with an empty one. Verified against the real function with per-node fake roots: NVMe on both nodes -> NVMe; on one node only -> NFS; on neither -> NFS; explicit MODEL_DIR and MODEL_PATH both bypass the probe; srun failing -> NFS. bash -n clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
We are about to run the same cards two ways -- STANDALONE (the pipeline sbatches the card's script) and MADENGINE (madengine generates the sbatch). Today the two agree only by coincidence, because each launcher hardcodes the site's facts separately. That drift has already cost real runs: run_multinode.slurm hardcoded the NFS model dir and never probed local NVMe, so every colocated run read its checkpoint over NFS while a full local copy sat on the node. Both siblings had probed NVMe for a while; counting m2m_nobackup across the three gave 5/6/1, and the odd one out was the bug. scripts/common/cluster.sh follows the pattern vLLM uses at .buildkite/amd-disagg/cluster.sh: sourced, every value defaulted with :-, so the layering is always "environment beats site default". That single property is what makes the two paths interchangeable, because both speak only environment variables -- STANDALONE through standalone_env.sh and sbatch --export=ALL, MADENGINE through deployment/slurm.py merging the card's env_vars into the sbatch it writes. cluster.sh only fills what neither set, so neither path changes. It also absorbs the two byte-identical copies of check_model_path, and drops a default of SLURM_SUBMIT_DIR that named one person's home directory. Card schema: the node count lived in one of two mutually exclusive places -- 39 cards had it only in args (the -N flag), which madengine never parses, and 4 had distributed.nnodes + slurm.nodes with args empty. madengine reads distributed.nnodes (deployment/slurm.py:677), so on that path the 39 fell back to a default and pyt_large_ep_bench_1n/_2n/_4n would all have run on the same node count. nnodes is DERIVED from the -N already in args, never invented; args is untouched, and the pipeline only consults nnodes when args is blank, so STANDALONE is unaffected. Verified locally, no cluster needed: - both-paths-agree: for all 43 multinode cards, the env STANDALONE produces and the env MADENGINE produces resolve identically through cluster.sh -- with no overrides, with image/model overrides, and with site overrides. - resolver replayed against per-node fake roots under both shell regimes (set -euo pipefail, and no flags at all): NVMe on all nodes picks NVMe; on a subset falls back to NFS; on neither is fatal; an empty staged directory loses to a working NFS copy; an explicit MODEL_PATH bypasses the probe; a distinct MODEL_DIR is honoured last; srun failing moves to the next candidate. - orphan-variable scan across all three launchers is clean, and re-introducing the COLOCATED_FORWARD_ENV bug makes it fail, so it would have caught d9a22b7. - every name a launcher stopped defining is provided by cluster.sh; the six that are not are function-local scratch from the deleted duplicates, with zero remaining references. - the -N in args equals distributed.nnodes equals slurm.nodes for all 43 cards; every models.json round-trips byte-exactly, so the diff is additive only. - bash -n on all four shell/slurm files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
benchmark_xPyD.sh sweeps CON="8 16 32 64 128 256 512" -- seven concurrencies.
Mixtral-8x7B, DeepSeek-V3 and DeepSeek-R1 still declared
decode.tp: --cuda-graph-bs $(seq 1 512), so the server captured 512 CUDA graphs
at startup and used seven of them.
Measured on a 2-node DeepSeek-V3 run in tp mode (DP_MODE=0, so decode.tp is the
line in play), from the decode node's own log:
Init torch distributed ends 30 s 5%
Load weight end 83 s 13%
Memory pool end 22 s 3%
Capture cuda graph end 512 s 79% <-- 511.58 s, 512 sizes
----------------------------------------
total to ready 651 s
Capture was four fifths of bring-up. The three sibling models at the top of this
file (Llama-3.1-8B, Qwen3-32B, Llama-3.3-70B-FP8) already ship exactly this seven
-value list in the same slot, and the decode.dp variants already use a short
explicit list too, so this makes the file internally consistent rather than
introducing a new convention.
Llama-3.1-405B-Instruct-FP8-KV keeps $(seq 1 8) deliberately: it is already small
and raising it has memory implications on a 405B model that this evidence does
not cover.
Safe for batch sizes that are not captured: sglang pads a running batch up to the
nearest captured size, which is why the seven-value list already works for the
three models using it.
Verified: every decode.tp capture list now equals CON exactly except the 405B
entry; models.yaml still parses; no $(seq 1 512) remains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unauthenticated remote command execution exposes workloads, while discarded process exit statuses and incorrect error accounting can cause failed runs to be reported as successful.
9 open issues
Unauthenticated agents expose remote command execution · New Result accounting can report failed runs as successful · New Headless worker failures are reported as successful exits · New Benchmark failure is discarded during cleanup · New Foreground launcher delays TERM cleanup trap · New Cleanup can kill unrelated sleep processes · New Search too shallow to include concurrency sweep results · New Resolved EP_TP_SIZE is not passed into Docker · New Barrier failure is ignored and launcher continues · New
What changed in this PR
This follow-up to #293 improves MAD’s multinode launcher reliability on Spur and shared nodes. The stacked changes also expand benchmark reporting, diagnostics, and standalone Kubernetes support.
Changes:
- Harden scheduler-path resolution, barrier handling, and container cleanup.
- Improve result collection, failure reporting, connector tuning, and model compatibility.
- Add offline checks and documentation for multinode workflows.
| File | Description |
|---|---|
scripts/vllm_multinode/serve_colocated.sh |
Add colocated serving, synchronization, benchmarking, and shutdown handling. |
scripts/vllm_multinode/models.json |
Register four Kimi-K3 colocated model cards. |
scripts/vllm_multinode/mad-config.kimi-k3.yaml |
Add layered configuration for the comparison card. |
scripts/vllm_dissag/tests/run_interactive.sh |
Split caches by role and forward additional benchmark settings. |
scripts/vllm_dissag/tests/run_all.sh |
Include additional offline suites. |
scripts/vllm_dissag/tests/parse_to_csv_assert.sh |
Check result parsing, failure status, and latency metrics. |
scripts/vllm_dissag/tests/jit_lock_assert.sh |
Check stale build-lock cleanup. |
scripts/vllm_dissag/tests/gpu_mem_trace_assert.sh |
Check GPU memory tracing and lifecycle handling. |
scripts/vllm_dissag/tests/gpu_clean_assert.sh |
Check pre-start GPU memory validation. |
scripts/vllm_dissag/tests/gate_check.sh |
Cover DeepSeek tensor parallelism with MoRIIO. |
scripts/vllm_dissag/tests/fault_capture_assert.sh |
Check fault capture and environment forwarding. |
scripts/vllm_dissag/tests/drive_cell.sh |
Forward additional interactive-run settings. |
scripts/vllm_dissag/tests/bench_model_name_assert.sh |
Check served-model names and prompt sizing. |
scripts/vllm_dissag/socket_barrier.py |
Add abort monitoring, timeouts, and fatal bind failures. |
scripts/vllm_dissag/README.MD |
Document model support, tuning, and standalone execution. |
scripts/vllm_dissag/gpu_mem_trace.sh |
Add host-side GPU memory-holder diagnostics. |
scripts/vllm_dissag/connectors/rixl.sh |
Honor recipe settings and improve startup failure handling. |
scripts/vllm_dissag/connectors/moriio.sh |
Apply transfer tuning and support tensor parallelism within expert pools. |
scripts/vllm_dissag/connectors/moriio.env |
Remove previously ineffective transfer defaults. |
scripts/vllm_dissag/benchmark_xPyD.sh |
Fix model naming, prompt sizing, and parser resolution. |
scripts/vllm_dissag/benchmark_niah.sh |
Resolve served names and publish retrieval results. |
scripts/vllm_dissag/benchmark_niah.py |
Increase answer budget and report truncation. |
scripts/vllm_dissag/benchmark_long_context.sh |
Publish results and honor shared benchmark settings. |
scripts/vllm_dissag/benchmark_agentic.sh |
Convert agentic aggregates into perf.csv. |
scripts/vllm_dissag/ARCHITECTURE.md |
Document Kimi-K3 worker topology and image details. |
scripts/sglang_disagg/tests/prr_flag_assert.sh |
Check compatibility with changed SGLang options. |
scripts/sglang_disagg/tests/parse_to_csv_assert.sh |
Check failed, missing, and empty-response results. |
scripts/sglang_disagg/tests/forward_env_assert.sh |
Check node-shape and fabric forwarding. |
scripts/sglang_disagg/tests/fault_capture_assert.sh |
Check GPU fault capture and watcher cleanup. |
scripts/sglang_disagg/socket_barrier.py |
Add abort monitoring, timeouts, and fatal bind failures. |
scripts/sglang_disagg/sglang_disagg_server.sh |
Avoid redundant installs and adapt recipe flags. |
scripts/sglang_disagg/sglang_disagg_mori_io_ep.sh |
Add coordinated failures, diagnostics, and option compatibility. |
scripts/sglang_disagg/sglang_compat.sh |
Adapt flags to installed SGLang capabilities. |
scripts/sglang_disagg/set_env_vars.sh |
Honor supplied InfiniBand device settings. |
scripts/sglang_disagg/README.MD |
Link standalone multinode instructions. |
scripts/sglang_disagg/parse_to_csv.py |
Report failed sweep cells and agentic metrics. |
scripts/sglang_disagg/mori_ep_env.sh |
Default MoRI’s GID index from NCCL’s setting. |
scripts/sglang_disagg/models.yaml |
Reduce tensor-parallel graph capture sizes. |
scripts/sglang_disagg/benchmark_xPyD.sh |
Fix iteration expansion and logged prompt counts. |
scripts/sglang_disagg/benchmark_agentic.sh |
Publish agentic aggregates as perf.csv. |
scripts/large-ep-benchmark/models.json |
Declare node counts and Slurm resources. |
scripts/common/verify_both_paths_generate.py |
Check standalone and madengine launcher equivalence. |
scripts/common/tests/launcher_verdict_assert.sh |
Check job verdicts, cleanup, ports, and forwarding. |
scripts/common/tests/cluster_fabric_assert.sh |
Check fabric detection and explicit overrides. |
scripts/common/k8s/tests/test_submit.py |
Check Kubernetes manifests, bundles, and result verdicts. |
scripts/common/k8s/tests/test_shim.py |
Check Kubernetes command stand-ins across agents. |
scripts/common/k8s/mad_k8s_node.sh |
Add pod initialization, launcher execution, and result publication. |
scripts/common/check_srun_quotes.py |
Detect unsafe quoting in embedded shell bodies. |
scripts/common/check_gpu_arch_declarations.py |
Check card and recipe architecture consistency. |
scripts/common/check_continuations.py |
Detect comments that break continued shell commands. |
README.md |
Link the documentation entry point. |
models.json |
Preserve the empty root registry; no semantic change. |
docs/README.md |
Add documentation navigation and a multinode glossary. |
docker/vllm_disagg_inference.ubuntu.amd.Dockerfile |
Pin rocSHMEM to a compatible revision. |
docker/sglang_disagg_inference.ubuntu.amd.Dockerfile |
Select architecture-specific stacks and bake in runtime dependencies. |
benchmark/kimi_k3/README.md |
Point MI300X users to multinode recipes. |
Review effort: Balanced
|
|
||
| if [ "$idx" != "0" ]; then | ||
| echo "[mad-k8s] node ${idx} (${MAD_K8S_SELF} on $(hostname)): serving tasks for ${MAD_K8S_NODE_PREFIX}-0" | ||
| exec python3 "$SHIM/mad_k8s.py" agent |
There was a problem hiding this comment.
Agreed, fixed in 5f4c642 (and in the same file in madengine #249, e4f2c83). The submitter now puts a random per-run token (MAD_K8S_AGENT_TOKEN, 256 bits) in every pod's environment. The agent will not start without it and rejects any request that doesn't carry it (constant-time compare). The head agent and the worker agents are covered the same way. I haven't added a NetworkPolicy: the pods use host networking, which NetworkPolicy doesn't cover. With the token, being able to reach the port is not enough to run anything. Only someone who can read the Job spec could get the token, and that person could already exec into the pods. Tests: a request with no token or another run's token is refused, and an agent with no token won't start. Note that #300 / #250 remove the agent entirely (each pod runs its own node), so this only matters for the spur-followup line.
| total = acct.get('records_total') or d.get('num_requests_total') or 0 | ||
| ok = d.get('num_requests_successful') or 0 | ||
| errors = acct.get('records_error_dropped') or 0 | ||
| max_err = float(os.environ.get('AGENTIC_MAX_ERROR_RATE', '0.10')) | ||
| invalid = os.path.exists(os.path.join(os.path.dirname(os.path.abspath(json_path)), 'RUN_INVALID.json')) | ||
| failed = ok == 0 or invalid or (total and errors / total > max_err) |
There was a problem hiding this comment.
Fixed in e0a233d: it now counts the way validate_agentic_result.sh does. With accounting, total = records_profiled + records_error_dropped. On the legacy schema, errors = total successful. Tests cover both.
| while _alive "${WORKER_PID}"; do | ||
| if [ -f "${SHUTDOWN_FLAG}" ]; then | ||
| echo "[colocated] worker ${NODE_RANK} got shutdown signal; stopping" | ||
| pkill -P "${WORKER_PID}" 2>/dev/null | ||
| kill "${WORKER_PID}" 2>/dev/null || true | ||
| break | ||
| fi | ||
| sleep 5 | ||
| done | ||
| wait "${WORKER_PID}" 2>/dev/null || true | ||
| exit 0 |
There was a problem hiding this comment.
Fixed in b50f90c. A worker that exits before the shutdown signal now fails with its own log tail and leaves a marker. The head reads the marker and fails right away. Tests are in colocated_worker_assert.sh.
| bash "${SHARED_DIR}/${BENCHMARK_SCRIPT_FILE:-benchmark_xPyD.sh}" | ||
|
|
||
| echo "[colocated] benchmark complete; stopping server" | ||
| pkill -P "${WORKER_PID}" 2>/dev/null; kill "${WORKER_PID}" 2>/dev/null || true | ||
| exit 0 |
There was a problem hiding this comment.
Agreed, fixed in 13ab833. The head now captures the benchmark's status, stops the server, and exits with that status. Test: a stand-in benchmark that exits 7 makes the head exit 7, and one that exits 0 makes it exit 0.
| cd "$(dirname "$script")" | ||
| echo "[mad-k8s] running $(basename "$script") ${MAD_K8S_ARGS:-} on ${SLURM_JOB_NODELIST}" | ||
| # shellcheck disable=SC2086 | ||
| bash "$(basename "$script")" ${MAD_K8S_ARGS:-} |
| kill -TERM "$_lp"; wait "$_lp"; _rc=$? | ||
| _is "$_rc" "143" "TERM during the run ends the job with 143" | ||
| _has "$(_stub)" "docker stop colocated_M_4242" "TERM during the run stops the containers" | ||
| pkill -f "sleep 30" 2>/dev/null |
There was a problem hiding this comment.
| # run that measured throughput, latency and cache hit reported "0 perf files". | ||
| # One set of rows per workload; see parse_to_csv.py. | ||
| if [ -n "${SLURM_JOB_ID:-}" ] && [ "${DRY_RUN:-0}" != "1" ]; then | ||
| mapfile -t _agg < <(find "/run_logs/${SLURM_JOB_ID}" -maxdepth 2 -type f \ |
There was a problem hiding this comment.
Correct, fixed in 3d81708 for both vLLM and SGLang. The search depth is now 3. With that change, rows from a sweep would have been named after the conc<N> directory, so parse_to_csv now names them by workload and concurrency ("agentic coding conc16"). Covered by a test.
| -e ROUTER_PORT=${ROUTER_PORT:-30000} \ | ||
| ${ROUTER_BINARY:+-e ROUTER_BINARY=$ROUTER_BINARY} \ | ||
| -e GPU_MEMORY_UTILIZATION=${GPU_MEMORY_UTILIZATION:-0.8} \ | ||
| ${GPU_MEMORY_UTILIZATION:+-e GPU_MEMORY_UTILIZATION=$GPU_MEMORY_UTILIZATION} \ |
There was a problem hiding this comment.
Correct, fixed in f9ecf75: the resolved value is passed with -e EP_TP_SIZE=$EP_TP_SIZE.
| echo "[colocated] waiting at container barrier on $(hostname -s)" | ||
| python3 "${SHARED_DIR}/socket_barrier.py" \ | ||
| --local-ip "${host_ip}" --local-port "${_BARRIER_PORT}" --enable-port \ | ||
| --node-ips "${IPADDRS}" --node-ports "${_BARRIER_PORT}" |
There was a problem hiding this comment.
Correct, fixed in 13ab833. A barrier that fails now fails the node with a message, before any server starts. Covered by a test.
cluster_job_verdict failed a run only when the perf CSV was missing or had a FAILURE row. A collector that found no metrics still writes the header, so such a run passed and madengine recorded "0 successful, 0 failed". When results are required, a CSV with no data rows is now "no parseable results", the same as a missing one; with results not required (SKIP_BENCHMARK) it still passes. launcher_verdict_assert: header-only fails when results are required and passes when they are not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agentic perf rows (vLLM and SGLang parse_to_csv.py) took the error rate over request_accounting.records_total, which includes the warmup records the aggregate drops on purpose. validate_agentic_result.sh counts errors over measured requests only (records_profiled + records_error_dropped), so with warmup the perf CSV could pass a run the validator rejects. Both now count as the validator: with request_accounting, ok = records_profiled and total = records_profiled + records_error_dropped; without it, the top-level counts. An aggregate that yields no metric rows now reports it on stderr; the header-only CSV it leaves fails the job verdict. The field names are those of the pinned InferenceX aggregate (utils/agentic/aggregation, ef8a17ec): request_accounting has records_total, records_profiled and records_error_dropped, and the top level num_requests_total / num_requests_successful; request_metrics.throughput has total / output / input tokens_per_second, plus per_gpu when the GPU count is known. parse_to_csv_assert (both): rows and SUCCESS for a normal aggregate, the validator's count with warmup, FAILURE at 15% errors over measured requests (10% over records_total), and no rows for an aggregate without metrics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker of the colocated launcher waited while its vllm serve was alive and then exited 0, also when the server had died on its own. Nothing told the head, whose server then never became ready: it waited out LOG_WAIT_TIMEOUT_SECONDS (4000 s), while the dead worker's task reported success. A worker whose server ends before the head's shutdown signal now prints the server's log tail, writes .worker_failed_NODE<rank> with its status in the shared per-job log directory, and exits 1. The head's readiness wait stops at once on such a marker, naming the worker; its exit trap raises the shutdown signal for the other workers, as before. The per-job log directory can be overridden with _RUN_LOGS, as in vllm_disagg.sh, for the new colocated_worker_assert (11): dying worker, head stopping on it, and a worker shut down by the head still exiting 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cluster_job_verdict failed a run only when the perf CSV was missing or had a FAILURE row. A collector that found no metrics still writes the header, so such a run passed and madengine recorded "0 successful, 0 failed". When results are required, a CSV with no data rows is now "no parseable results", the same as a missing one; with results not required (SKIP_BENCHMARK) it still passes. launcher_verdict_assert: header-only fails when results are required and passes when they are not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agentic perf rows (vLLM and SGLang parse_to_csv.py) took the error rate over request_accounting.records_total, which includes the warmup records the aggregate drops on purpose. validate_agentic_result.sh counts errors over measured requests only (records_profiled + records_error_dropped), so with warmup the perf CSV could pass a run the validator rejects. Both now count as the validator: with request_accounting, ok = records_profiled and total = records_profiled + records_error_dropped; without it, the top-level counts. An aggregate that yields no metric rows now reports it on stderr; the header-only CSV it leaves fails the job verdict. The field names are those of the pinned InferenceX aggregate (utils/agentic/aggregation, ef8a17ec): request_accounting has records_total, records_profiled and records_error_dropped, and the top level num_requests_total / num_requests_successful; request_metrics.throughput has total / output / input tokens_per_second, plus per_gpu when the GPU count is known. parse_to_csv_assert (both): rows and SUCCESS for a normal aggregate, the validator's count with warmup, FAILURE at 15% errors over measured requests (10% over records_total), and no rows for an aggregate without metrics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker of the colocated launcher waited while its vllm serve was alive and then exited 0, also when the server had died on its own. Nothing told the head, whose server then never became ready: it waited out LOG_WAIT_TIMEOUT_SECONDS (4000 s), while the dead worker's task reported success. A worker whose server ends before the head's shutdown signal now prints the server's log tail, writes .worker_failed_NODE<rank> with its status in the shared per-job log directory, and exits 1. The head's readiness wait stops at once on such a marker, naming the worker; its exit trap raises the shutdown signal for the other workers, as before. The per-job log directory can be overridden with _RUN_LOGS, as in vllm_disagg.sh, for the new colocated_worker_assert (11): dying worker, head stopping on it, and a worker shut down by the head still exiting 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ils the node The head discarded the benchmark's exit status and always exited 0, so a run whose benchmark failed reported success. It now stops the server and exits with that status. A container barrier that fails (a node's container never came up) used to be ignored and the node went on to start its server; it now fails the node with a message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sweep over several concurrencies writes each aggregate JSON under
<workload>/conc<N>/, one level deeper than the -maxdepth 2 search reached, so a
sweep published no perf.csv. Search to depth 3, and name each row by workload
and concurrency ("agentic coding conc16") instead of by the conc<N> directory
alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It resolved EP_TP_SIZE from the recipe for the host-side JIT cache split but did not forward it, so the in-container launcher could run with a different value than the cache was keyed on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pkill -f "sleep 30" could kill any process on the machine running that command. The stand-in now sleeps a duration unique to the test run, and only that exact command is killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ils the node The head discarded the benchmark's exit status and always exited 0, so a run whose benchmark failed reported success. It now stops the server and exits with that status. A container barrier that fails (a node's container never came up) used to be ignored and the node went on to start its server; it now fails the node with a message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sweep over several concurrencies writes each aggregate JSON under
<workload>/conc<N>/, one level deeper than the -maxdepth 2 search reached, so a
sweep published no perf.csv. Search to depth 3, and name each row by workload
and concurrency ("agentic coding conc16") instead of by the conc<N> directory
alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It resolved EP_TP_SIZE from the recipe for the host-side JIT cache split but did not forward it, so the in-container launcher could run with a different value than the cache was keyed on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pkill -f "sleep 30" could kill any process on the machine running that command. The stand-in now sleeps a duration unique to the test run, and only that exact command is killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The file holding forwarded values (which can include credentials) had a predictable /tmp name and was created with rm/truncate/chmod, so another user could pre-create it or a link at that name. mktemp creates it new with mode 600, and it is removed however the node exits, a cancelled run included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The file holding forwarded values (which can include credentials) had a predictable /tmp name and was created with rm/truncate/chmod, so another user could pre-create it or a link at that name. mktemp creates it new with mode 600, and it is removed however the node exits, a cancelled run included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cancelled run's other ranks can start their stand-in container after the test's one-time cleanup; the test's EXIT trap now kills them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cancelled run's other ranks can start their stand-in container after the test's one-time cleanup; the test's EXIT trap now kills them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent on every pod runs any command sent to its port, and with host networking anyone who could reach that port could run commands in the pod. The submitter now puts a random per-run token (MAD_K8S_AGENT_TOKEN) in every pod's environment; the agent refuses to start without one and refuses any request that does not carry it. Same files as madengine's k8s_shim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pod 0 ran the card's script in the foreground, so a TERM from Kubernetes ran the trap only after the script had finished on its own (or the pod was killed). The script now runs in the background and TERM is forwarded to it, so its teardown runs, then the peers are released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The _way4 card carries only MAD_CONFIG; its MODEL_NAME and the rest are in mad-config.kimi-k3.yaml, which madengine resolves and plain sbatch does not. Submitted with sbatch, the launcher stopped on "set MODEL_NAME" with no hint why. It now stops before any work and says the file is a madengine layered config. Tested in launcher_verdict_assert.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings ROCm#242's review fixes into ROCm#293: the agentic error rate counted as validate_agentic_result.sh counts it, sweep (conc<N>) aggregates in perf.csv, the colocated head's benchmark status and barrier failures, a failed worker failing the job, EP_TP_SIZE forwarded by run_interactive.sh, header-only perf CSVs failing the verdict, the mktemp forwarded-env file, and the MAD_CONFIG refusal in the colocated launcher. Conflicts were the same fixes applied on both sides: kept this branch's cluster_expand_nodelist, CONTAINER_ENV/GPU_MEM_TRACE and long-context tests, and took ROCm#242's launcher directory resolution, header-only verdict and test-sleep cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent on every pod runs any command sent to its port, and with host networking anyone who could reach that port could run commands in the pod. The submitter now puts a random per-run token (MAD_K8S_AGENT_TOKEN) in every pod's environment; the agent refuses to start without one and refuses any request that does not carry it. Same files as madengine's k8s_shim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5f4c642)
…the container It mounted a caller-set JIT_CACHE_HOST but did not pass the variable in, so _clear_stale_jit_locks in vllm_disagg.sh took the cache for the job's own and removed every AITER lock, including fresh ones of another job sharing the cache. Both values now go in as the batch launchers pass them. Checked in jit_lock_assert.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The _way4 card keeps MODEL_NAME and its other values in MAD_CONFIG, which madengine resolves. submit.py reads only env_vars, so its pods started without them and the launcher stopped on "set MODEL_NAME". It now refuses such a card (MAD_CONFIG, or a mad-config.yaml next to the script) and says madengine is needed. Tested in test_submit.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alue The per-architecture stack left `ARG BASE_DOCKER=` empty and chose the base through stage names. Tools that read the base image from that ARG got nothing: the mad-validate workflow's Dockerfile renderer stopped with "no base image ... could not read ARG BASE_DOCKER from docker/sglang_disagg_inference.ubuntu.amd.Dockerfile", and tools/utils.py get_base_docker returned "". BASE_DOCKER now holds the gfx942 base (the default architecture, the same image as before the split) and BASE_DOCKER_GFX950 the gfx950 one; MAD_SYSTEM_GPU_ARCHITECTURE still picks between them. Checked with `docker buildx build --check` for both architectures and by rendering every template on a merge with develop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit put the JIT_CACHE_HOST and JIT_LOCK_STALE_MIN options on the EP_TP_SIZE line. Same options, one per continuation line like the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings ROCm#293's head: ROCm#242 (through its merge into ROCm#293), the run_interactive JIT_CACHE_HOST / JIT_LOCK_STALE_MIN forwarding, the standalone k8s refusal of a layered-config card, and the SGLang disagg Dockerfile's per-architecture BASE_DOCKER / BASE_DOCKER_GFX950 ARGs that the mad-validate workflow's renderer reads. Conflicts: kept this branch's disagg barrier-port test and cache directories in launcher_verdict_assert.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


Follow-up to #293 (stacked on
spurupdate; until #293 merges this PR also shows its commits). Three fixes found running the disaggregated launchers on a Spur cluster and on shared nodes:sbatch. The SGLang and colocated launchers took their directory from the script's own path.sbatchruns a copy of the script from the scheduler's spool directory (SLURM's slurmd, Spur's/var/spool/spur), so they looked forcommon/cluster.shthere and stopped at once (Error: missing /var/spool/spur/common/cluster.sh). madengine runs the script in place, so it never hit this. Both now resolve their directory like the vLLM launcher: the script's own if it holds the launcher's files, else$SLURM_SUBMIT_DIR/scripts/<launcher>,$SLURM_SUBMIT_DIR, or the working directory. All 22 SGLang and colocated cards give the same run from a spooled copy as in place.fuser -kon every node, killing whatever else listened there, and both disagg launchers began withdocker ps -q | xargs docker stop, which stops other users' containers on a shared node. Barrier ports are now21000 + job id mod 8000(clear of the fixed ports the launchers and engines use), nothing is killed on a port, and start-up cleanup stops this launcher's leftover containers only, by name.launcher_verdict_assert.shno longer creates a JIT cache outside its temp directory.Tests:
launcher_verdict_assert.shruns the vLLM disagg launcher against stand-ins and checks the per-job ports, that nothing is killed on a port, and the name filter.