Skip to content

Hardware generalization: recipes declare what they need, one card runs on any GPU - #317

Open
Cemberk wants to merge 216 commits into
ROCm:developfrom
Cemberk:hw-generalization
Open

Cemberk wants to merge 216 commits into
ROCm:developfrom
Cemberk:hw-generalization

Conversation

@Cemberk

@Cemberk Cemberk commented Nov 19, 2026

Copy link
Copy Markdown

Stacked on #300 (DI refactor) and #295 (image specs). Until they merge, this diff includes them.
Review from commit 3be2b6f on: 30 commits, plus the merge of #295.

What this PR adds on top of #300

In #300 a recipe was a set of hand-written engine flags for one GPU, and a card could run only where its GPU_ARCHS / skip_gpu_arch allowed. GPU count, NIC family, rails, GID and GPU arch were hard-coded in cluster.sh, the connectors and the .env files. This PR turns those facts into data the launchers check and use. One card can then run on whatever GPU it lands on.

  1. Recipes say what each role needs, and the launcher checks it (3be2b6f, 36e5028, 7fab1ad, 215c94a)
  • compat/rocm.yaml gains a capability catalog:
    • formats;
    • accelerators: gfx942, gfx950, CDNA 5;
    • kernels per @, meaning the formats each engine build runs and converts on each GPU;
    • engine arguments.
  • Recipes in models.yaml gain roles: (prefill, decode, colocated). For each role they give weight, activation and KV formats, an optional per-role checkpoint, and weight placement.
  • New gate cluster_check_recipe runs in all three launchers after the GPU gate:
    • It resolves each role against the GPU and the image's own build (MAD_PIN_SET from the card's PINS).
    • It refuses with the unmet need, e.g. FP4 activations on gfx942.
    • It adds the arguments the flags lack, for example the int4 conversion on gfx942 or the KV-cache dtype. These reach the engines as MAD_ROLE_ENGINE_ARGS_.
    • A role with its own checkpoint serves it under the model's served name.
    • RECIPE_CHECK=0 skips the gate. Recipes without roles behave as before.
  • Every evidenced vLLM and SGLang recipe now declares its roles. A test checks every card: its recipe must resolve on exactly the GPUs the card runs on.
  1. One recipe, a variant per GPU, sized from the GPU (e4cb1dc, 45fbae7, 4d76ce1, 4c5ee26)
  • New scripts/common/recipe/models.py is the single recipe reader. It adds:
    • variants: per accelerator or platform, plus default_arch;
    • alias: recipes;
    • memory: {fraction: 0.1}, which sizes a value from the GPU's memory.
  • The launcher detects the GPU and applies that GPU's variant. It writes the job's resolved recipe to MAD_RECIPE_FILE so every container runs the same one.
  • The two Kimi-K3-MXFP4 recipes are now one recipe with gfx942 and gfx950 variants. The effective recipes are identical to before.
  • requires: {min_gpu_memory_gb} and accuracy: records are new.
  1. No GPU count, NIC family, arch or site in code (bfacd8b, 90088d1, 33c30ff, 2890d35)
  • The catalog now also describes:
    • platforms: mi300x-oam-8, mi325x-oam-8, mi355x-ubb-8, mi455x-helios-tray;
    • fabrics: cx7, ainic, thor2, with detection rules;
    • scale-up fabrics;
    • example site profiles (MAD_CLUSTER, or MAD_CLUSTER_FILE kept outside the repo).
  • The changes in code:
    • cluster.sh takes the fabric from platform/fabric.py (the case blocks are removed) and GPUS_PER_NODE from the platform.
    • cluster_verify_platform refuses nodes that don't match the named platform.
    • The connectors and the SGLang MoRI env read MAD_FABRIC_* from the launcher instead of one site's literal rails, GID and interface.
    • MORI_GPU_ARCHS=gfx942 is removed from the .env files.
  • Every value MAD derives is only a default: a value passed through config wins, and config.resolved.json records where it came from. passthrough_assert.sh covers each one.
  1. Topology spec v2 (230e511, c7a25ab)
  • New in the spec:
    • pools sized in GPUs on a platform, plus compute partitions;
    • role kinds: draft, encoder, embedding, kv_cache;
    • kv_format checked at the prefill→decode handoff;
    • a replicas parallel mode;
    • placement (same_domain / spread_domains), checked against scale-up domains.
  • Scheduler hints: --segment on SLURM; pod affinity on Kubernetes through k8s/submit.py.
  • v1 specs resolve exactly as before.
  1. Best-known configuration is the default run (8cb6b8b, af17152, 162b48b)
  • New perf/model.py: a roofline model and a layout search. --emit-spec writes the best layout as a v2 topology spec.
  • New perf/registry.py with record, best and promote.
  • perf/best_known.jsonl records are laid over the recipe by default, the same way in STANDALONE and madengine runs:
    • MAD_BEST_CONFIG=0 runs the recipe as written.
    • Config still wins.
    • The file ships empty, so today's runs are unchanged.
  1. New serving types and benchmarks (2b4d651, 1bab2a6, 7733d13, 88b72dc, 20ae121, 5189e82)
  • Speculative decoding (speculative:) and multi-LoRA (lora:) as role needs, mapped to each engine's flags.
  • REPLICAS=1 runs independent servers per node behind replica_router.py.
  • An embedding role.
  • New benchmarks: BENCHMARK_SCRIPT=accuracy, embeddings, lora.
  • serving_types.yaml lists what is runnable. A test holds it to what the launchers accept.
  1. Bring-up and submit-side checks (186b6f2, 8a1b430, a7ae3c2)
  • pins.py validate refuses pin sets for unknown GPUs.
  • recipe.py bringup --arch cdna5 reports what a new GPU still lacks.
  • recipe/precheck.py lets a submitter (madengine) refuse a recipe before it queues.
  • New doc on adding an accelerator, NIC, site, format or serving type as data.
  1. GPU-agnostic cards and end-to-end examples (855a843, 7fd810e)
  • 5 cards tagged any_gpu, for Kimi-K3 colocated and disaggregated. On gfx942 they run the same engine commands as their MI300X-named twins. The GPU-named cards are unchanged.
  • scripts/common/workloads/: a library of inherited workload layers (defaults:) for base, serving shapes, benchmarks and model families.
  • 8 e2e_example cards built on it: DeepSeek-V3 MTP and disaggregated quantization, Kimi-K3 NIAH and on Helios, Llama-405B TP8×PP2, Qwen3 replicas with LoRA, Qwen3 embeddings, and DeepSeek-R1 on SGLang. Documented in docs/e2e-examples.md.
  1. MAD runs a card's workload file without madengine (15024f7)
  • config/workload.py (standard library and PyYAML) reads MAD_CONFIG / mad-config.yaml. The launchers apply it at submit level, and k8s/submit.py uses it when madengine is absent.
  • A test checks that madengine's reader gives the identical environment.

Fixes found along the way:

  • In config_apply_recipe, a quoted python3 -c body dropped the recipe layer whenever a best-known record matched. check_srun_quotes now checks python3 -c bodies too.
  • The recipe check could add the same injected arguments twice. Fixed.
  • Launcher model allow-lists now follow recipe aliases.

Changes to existing cards' runs

  • The engine commands are unchanged for all 55 cards in all 5 dispatch modes.
  • The goldens change only in forwarded env:
    • -e MAD_FABRIC_* with the same values as the old literals;
    • -e MAD_RECIPE_FILE on the Kimi-K3-MXFP4 cards;
    • -e GPUS_PER_NODE on SGLang;
    • the duplicate MORI_GPU_ARCHS is removed.

Testing (offline; real-cluster runs follow)

  • Unit tests pass: recipe, topology, config, k8s, perf, pins, accuracy, replicas, embeddings, LoRA, workload.
  • Launcher assert suites pass: config layers 38, recipe check 29, platform 15, passthrough 15, and the rest.
  • Snapshots: 55 cards × 5 dispatch modes, 0 differ.
  • E2E examples pass on each target GPU, and they pass without madengine installed.
  • Pipeline path equivalence, STANDALONE vs MADENGINE: SLURM 67 cards and Kubernetes 67 × 2, both 0 mismatches.

Cemberk and others added 30 commits August 13, 2026 17:17
… selector

The xPyD harness already models pools that span multiple nodes (xP/yD count
NODES, roles are MASTER/CHILD via --data-parallel-start-rank + --headless), but
the wideEP path hardcoded TP=1: dp_size was xP*GPUS_PER_NODE and the connector
emitted a literal `-tp 1`. That makes any model whose REPLICATED (non-expert)
weights exceed one GPU inexpressible - e.g. Kimi-K3 on MI300X, where TP1/DP16
needs 190.7 GiB/GPU (> 192 GB HBM) and TP2 is required to fit.

- vllm_disagg.sh: topology math is TP-aware. dp_per_node = GPUS_PER_NODE/TP_SIZE
  feeds dp_size, dp_size_local and both start-rank calculations. TP_SIZE defaults
  to 1 and is validated to divide GPUS_PER_NODE.
- moriio.sh: emit -tp ${TP_SIZE}; clamp --api-server-count to dp_size (it must be
  <= data-parallel-size or the frontend DP balancer routes to ranks that do not
  exist - only reachable once TP>1 shrinks dp_size below GPUS_PER_NODE).
- moriio.sh: advertise the peer pool's node IPs as moriio_pod_hosts when a pool
  spans >1 node. Without it the connector falls back to the peer MASTER only and
  KV writes aimed at ranks on a peer CHILD node silently miss, so those ranks
  decode with no context. Emitted only when xP>1 || yD>1.
- run_xPyD_models.slurm: forward TP_SIZE/GPUS_PER_NODE, wire BENCHMARK_SCRIPT=niah
  to the already-present benchmark_niah.sh, forward NIAH_* knobs.

Default-neutral: at xP=1 yD=1 TP_SIZE=1 - the topology every registered entry
uses - DRY_RUN output is byte-identical to develop across all four node roles.
At xP=2 yD=2 TP_SIZE=2 it emits tp=2, dp_size=8, dp_local=4, child start-rank 4
(= EP16 per pool over 16 GPUs).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…Kimi-K3 recipe

Two parts.

1) moriio.sh wideEP dropped ${model_args[@]} on the floor. models.yaml documents
   per-model dp: tuning as supported ("both connectors now append it") and rixl's
   deepep path does append it, but the moriio wideEP branch parsed the flags,
   logged them, and then emitted an argv without them. Verified with a sentinel
   flag: present in the log line, absent from the command. Now appended, matching
   rixl. Every existing wideEP model has an empty dp: block, so emitted argv is
   unchanged for all of them.

2) Kimi-K3 recipe, entirely as data: env: block for the gfx942 knobs
   (VLLM_ROCM_USE_AITER_MLA=0 - the AITER MLA kernel is gfx950-only -
   AITER_SITUV2_A8W4=1 for the packed-int4 SiTUv2 MoE path, KDA conv state layout,
   40e9 KV cache for >600K contexts, per-role cudagraph + MoRI backends), and a
   dp: block for the serve flags the connector does not emit (reasoning parser,
   1M max-model-len, batched-tokens pinned at 2048, MoE quantization-config).
   Registered in MORI_EP_VALID_MODELS and WIDE_EP_ONLY_MODELS.

   The JSON quantization-config carries its own shell quotes: the launcher
   word-splits dp: via eval, which would otherwise strip the JSON's double quotes
   and hand vLLM invalid JSON. Verified the emitted value parses as JSON.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker/pyt_vllm_kimi_k3_mi300x.ubuntu.amd.Dockerfile
  The shared vllm_disagg_inference stack with the pins K3 needs on gfx942
  (MoRI 1.2.2, AITER 0.1.19 + flydsl 0.2.4, the K3+MoRIIO vLLM commit) plus the
  K3-aware AITER graft from the public vendor image - without that graft the K3
  MoE profiling shape finds no tuned FlyDSL config, falls back to a heuristic
  kernel and aborts LLVM inside determine_available_memory.

  Differences from the recipe this is ported from:
  - every source pinned to an immutable SHA, not a fork branch name (branches on
    personal forks can be force-pushed; MAD needs the image rebuildable later)
  - GH_TOKEN build-arg dropped: all three repos are public, and a token passed
    this way is recorded in image metadata
  - the flydsl 0.2.4 re-pin happens at build time, so the launcher no longer runs
    pip install inside every container at serve time
  - WITH_NIXL defaults to 0 (K3 uses the moriio connector only)
  - no runtime patchers: all connector/KDA fixes are committed in the pinned vLLM

scripts/vllm_multinode/
  Colocated (single-instance) multi-node serving - the counterpart to
  vllm_dissag, for models that do not fit one node but want lowest single-request
  latency rather than disagg throughput. TP within a node, PP across nodes.

  It owns only node discovery, the container launch and the head/worker split;
  models.yaml, socket_barrier.py, benchmark_xPyD.sh, benchmark_niah.sh and
  parse_to_csv.py are reused from vllm_dissag (scripts/ is mounted whole), so the
  model recipe and the CSV pipeline have one home.

  The three K3 colocated variants collapse to this one script: dry-run confirms
  pp2xtp8 / allgather / moriep argv differ only by --enable-expert-parallel and
  --all2all-backend, so they become three models.json entries rather than three
  near-identical run.sh copies.

Not built here - this environment has no docker and one 8-GPU node; the harness
is verified by DRY_RUN argv inspection only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
models.json: four entries, all skip_gpu_arch gfx950 (the existing four Kimi-K3
entries are gfx950/TP8 single-node and all carry skip_gpu_arch gfx942, so K3 is
currently skipped entirely on MI300X - this is that gap):

  pyt_vllm_disagg_mori_kimi-k3               -N 4  xP2 yD2 TP2 -> EP16/pool
  pyt_vllm_kimi-k3_mi300x_pp2xtp8            -N 2  TP8 x PP2, no EP
  pyt_vllm_kimi-k3_mi300x_wideep_allgather   -N 2  + EP, allgather_reducescatter
  pyt_vllm_kimi-k3_mi300x_wideep_moriep      -N 2  + EP, mori_low_latency

The three colocated entries differ only in ENABLE_EP / ALL2ALL_BACKEND /
AITER_SITUV2_A8W4, which is why they share one launcher.

Result reporting: BENCHMARK_SCRIPT=niah previously produced logs and nothing
else, so a model declaring multiple_results would have reported no metric.
- parse_to_csv.py gains --niah: parses benchmark_niah.py output and writes the
  same 29-column madengine perf.csv as the throughput path (verified identical
  column sets). One row per context size, performance = needles found /10. A
  size that errored is written as a FAILURE row with performance 0 rather than
  dropped, so a pass->crash regression is visible instead of silent.
- benchmark_niah.sh calls it.
- Both slurm launchers copy /run_logs/$SLURM_JOB_ID/perf.csv to
  ./perf_$MODEL_NAME.csv on completion: madengine resolves multiple_results
  against the job directory it launched from, not the container log mount.
  No-op for entries that do not declare multiple_results.

Verified: madengine discover goes 133 -> 137 with exactly these four added and
none removed; every referenced dockerfile and script exists; -N matches each
topology; the int4 quantization-config survives shell word-splitting as valid
JSON; DeepSeek moriio and rixl/deepep DRY_RUN output is byte-identical to
develop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ports the writeups from PR ROCm#193 next to the existing gfx950 K3 doc, keyed to the
madengine tags rather than to shell scripts: topology rationale (why MI300X needs
multi-node and why the disagg pool needs TP2), the gfx942 knobs and where each
one lives now, the NIAH results, and the three root-cause fixes.

Corrections carried in:
- the NIAH ceiling is 900K throughout. The upstream READMEs still said 300K in
  seven places after two later commits raised it to 500K and then 900K.
- states that the NIAH harness sizes context in WORDS (~1.3x tokens) while the
  results table is in tokens, so the two columns are not comparable.

benchmark/kimi_k3/README.md gains a pointer: all four entries there carry
skip_gpu_arch gfx942, which is exactly the gap the MI300X page fills.

Every tag, node count and knob value in the new page is checked against
models.json and models.yaml.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…metadata

The four MI300X Kimi-K3 entries could not run correctly on a cluster. Three
separate defects, all from the colocated launcher inheriting conventions that
only hold for the disaggregated one.

Node sizing. The entries carried `"args": "-N 4 -n 4"` on the assumption that
madengine forwards args to sbatch. It does not — args are appended to
`bash <model>.slurm`, and neither launcher parses $@, so the flags were dropped
and the job was submitted with the `slurm.nodes` default of 1. Sizing now uses
`distributed.nnodes`, which madengine reconciles into `#SBATCH --nodes`, and the
inert args are removed. `slurm.nodes` is deliberately left unset: setting it
selects the multi-node preset, whose `NCCL_SOCKET_IFNAME=eth0` is forwarded into
the container by run_xPyD_models.slurm and would override the fabric interface
on an RDMA cluster. `slurm.time` is set explicitly instead, since the
single-node preset's 12 h default is short for a full NIAH sweep.

NIAH model tag. benchmark_niah.sh requested MODEL_PATH as the model name,
correct for the disagg path where vLLM defaults served_model_name to the serve
argument. serve_colocated.sh passes `--served-model-name "$MODEL_NAME"`, so
every request 404'd and all three colocated entries — which default to
BENCHMARK_SCRIPT=niah — would have recorded a full sweep of FAILURE rows. The
harness now honors NIAH_MODEL when a launcher sets it, and the colocated
launcher sets it to the tag it actually serves.

Colocated run metadata. parse_to_csv derived nnodes, n_gpus, deployment_type and
tags from xP/yD. serve_colocated.sh exports xP=1 yD=0 only so the shared log
filenames stay unique, so a 2-node 16-GPU colocated run reported itself as a
1-node 8-GPU `disagg_1P0D` with a nixl backend it never used. Node count now
comes from NNODES (exported by both launchers), and a launcher whose shape is
not "xP prefill + yD decode" states its own identity via PERF_DEPLOYMENT_TYPE /
PERF_TAGS. The disagg path is byte-identical to before.

Also corrects the README, which documented the args-to-sbatch behavior that does
not exist and a quick start that cannot work now that the "<supply-your-image>"
placeholder is rejected rather than pulled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NUM_NODES is derived from the requested topology (xP+yD), not from the
allocation, and the node list was then truncated with `head -n $NUM_NODES`. A
short allocation therefore produced a silently wrong topology: pools were built
from nodes that were never allocated, and the run failed much later as a
connector handshake timeout with no indication of the real cause.

This mattered little while these models were always launched from a matching
`salloc`, but the sbatch path now sizes the allocation from the model card, so a
mismatch between card and cluster is a reachable state. Check up front and name
the three ways to fix it.

The colocated launcher already fails on the equivalent mismatch via its
TP*PP == NNODES*GPUS check, so it needs no counterpart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ntract

parse_to_csv.py hand-wrote a full 29-column perf.csv, assembling node counts, GPU
counts, launcher, image and tags from the environment. Most of that is not the
workload's to know: madengine already owns it, and the guesses were wrong for the
colocated launcher, which sets xP=1 yD=0 only to keep log filenames unique.

The NIAH writer now emits a narrow CSV — model, performance, metric, status, plus
descriptive columns — and madengine merges in the run metadata via the entries'
`multiple_results` declaration. The columns that genuinely describe the workload's
own configuration rather than its placement (tp, pp, ep_backend, prefill_decode)
move to descriptive columns, matching how scripts/vllm/run_vllm.py already reports
tp/dtype/bs on the templated path. Nothing is lost; it is relocated to whichever
side actually knows it.

`status` stays explicit: an errored context size scores 0, and deriving status from
performance would file that real failure as a SUCCESS.

Scoped to leave every other workload alone. The NIAH writer is reachable only by the
four Kimi entries, which all declare `multiple_results`. The throughput writer keeps
its full-schema output by default, because the twelve disagg cards that use it
declare no `multiple_results` and madengine reads their CSV directly with no
metadata to merge — a narrow CSV there would drop every descriptive column. Their
output is byte-identical; `--narrow` is the opt-in for migrating one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…uted.nnodes

madengine reads only slurm.nodes when emitting `#SBATCH --nodes`
(deployment/slurm.py: `self.nodes = slurm_config.get("nodes", 1)`), and
nothing maps distributed.nnodes across — build_orchestrator copies nnodes
into deployment_config.distributed for launcher detection only. The four
MI300X model cards carried nnodes but no slurm.nodes, so every one of them
would have been submitted as a 1-node job. The launcher's
TP*PP == NNODES*GPUS_PER_NODE assertion catches it, but only after the
allocation is granted.

Add nodes/gpus_per_node to the four slurm blocks, and ship the site configs
that carry the values a model card cannot:

  - results_dir, absent from the key list madengine copies out of models.json,
    and required because the slurm_multi collector globs results_dir for
    perf*.csv rather than resolving multiple_results
  - MODEL_DIR / LOG_PATH, which otherwise default to /shared_inference

*.json is gitignored repo-wide, so the templates need explicit negations or
they never reach a clone.

Correct the two README claims that did not match madengine's behavior
(nnodes-sizes-the-allocation, multiple_results-is-resolved-first), document
that --account/--qos are unwired for slurm_multi and need SBATCH_ACCOUNT,
and drop --keep-model-dir from the SLURM examples — it is a local-Docker
flag that madengine ignores with a warning on SLURM.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four defects, each found by running the recipes end to end on a real
MI300X SLURM cluster for the first time. None of them could have survived
a single successful run, which is consistent with these cards never having
been executed through madengine.

1. Build was not reproducible. Every source is pinned to a commit SHA, but
   pip resolves build dependencies in an isolated environment from PyPI at
   build time, so nothing pinned the toolchain. setuptools >= 80 added
       assert isinstance(self.compiler, CCompiler)
   to distutils' build_ext.build_extension, which MoRI's legacy
   Cython.Distutils.build_ext path violates, so the amd_mori wheel failed
   with "AssertionError: run() must precede build_extension()" while every
   pinned SHA was still correct (job 223849). PIP_CONSTRAINT is the only
   mechanism that reaches inside pip's build isolation -- installing
   setuptools in the image does not. Set globally so the AITER, vLLM and
   router stages cannot regress the same way. Build then completed in
   19 minutes, all 18 stages (job 223872).

2. RDMA library mounts used -e, which is true for directories. Docker
   creates a DIRECTORY at a bind-mount source that does not exist, so the
   first run on a node lacking e.g. libionic.so.1 leaves an empty directory
   behind, and every later run on that node mounts a directory over a file
   inside the image:
       OCI runtime create failed: ... not a directory
   Self-propagating and per-node, so it presents as intermittent. The
   surviving node then waits at socket_barrier.py forever for a peer that
   already died -- 4h10m of "Waiting for nodes. . ." before the wall clock
   (job 223909). Use -f so only regular files are mounted.

3. pyt_vllm_kimi-k3_mi300x_pp2xtp8 omitted the gfx942 MoE requirement that
   the two wideep variants carry. gfx942 has no scaled-MXFP4 MFMA and the
   a16w4 SiTUv2 heuristic FlyDSL kernel cannot codegen there, so the MoE
   must be requantized to packed int4. Without it the worker aborts inside
   determine_available_memory with
       LLVM ERROR: Do not know how to expand this operator's operand!
   and quantization_config=None in the engine config (job 224132). This is
   a hardware requirement, not an EP-specific tuning, so all three
   colocated cards now carry AITER_SITUV2_A8W4=1 and the int4 config.

4. A run that produced no results exited 0. The launchers warned about a
   missing perf CSV and returned success anyway, so madengine recorded
   exit_code=0 in its completion marker and the collector reported
   "0 successful, 0 failed" -- a hard engine crash was indistinguishable
   from a clean run. For a benchmark repo that is the worst possible
   reporting outcome. Both launchers now exit 1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of these cost hours to diagnose on a real cluster, and in every case
the symptom is a long way from the cause -- an intermittent multi-node
hang that is actually a poisoned bind-mount path, a wheel build that fails
while every pinned SHA is correct, an LLVM abort that is a missing
quantization flag. Record them so the next person reads instead of
rediscovers.

Also record the cluster facts a model card structurally cannot carry: the
SLURM association wall-time cap (which silently parks a job as PENDING
rather than failing), account requirements, docker on compute nodes, and
the fact that a registry-less cluster has no supported path to distribute
an image -- --build-on-compute requires --registry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds Kimi-K3-MXFP4 (MI300X gfx942 MXFP4 MoE) to the vllm_dissag framework:
- models.yaml: K3 recipe env anchor + model entry (TP2×DP8, MoRI-EP)
- run_xPyD_models.slurm: allowlists + 2P/2D validation + JIT cache split
- moriio.sh: K3-gated topology (pod_hosts, api-server-count, headless KV)
- vllm_disagg.sh: pod hosts export + _model_config_to_array helper
- argv_assert.sh: K3 offline tests

Config-only integration; image build documented in README.

Co-Authored-By: Claude <noreply@anthropic.com>
…lsifiable

Both found on the first end-to-end MI300X run (job 224200), which served
Kimi-K3 and produced results -- but results that could not be trusted, from
a job that then would not exit.

Deadlock on teardown. serve_colocated.sh has the head kill only its own
server and exit, while every worker sits in `wait` on a headless vLLM that
nothing stops. srun waits on those tasks, so the job holds all nodes until
the wall clock: 4.5 hours of two idle exclusive MI300X nodes AFTER the
benchmark had already written perf.csv. The head now raises a sentinel on
the shared log volume and workers watch for it. The sentinel is raised from
an EXIT trap so a head that fails or times out cannot strand workers either.

NIAH could not distinguish truncation from retrieval failure. Scoring reads
content + reasoning_content, and K3 is a reasoning model served with
--reasoning-parser kimi_k3, so a trace that exhausts max_tokens is cut off
and only the earliest needles survive. At the old 2048 default, two of four
context sizes scored 1/10 and both listed exactly ANIMALS[0] -- the needle
placed first in the haystack. That signature is a truncated trace, not a
model failure, but finish_reason was never recorded so the two were
indistinguishable, and every row was written SUCCESS regardless.

  - record finish_reason and print it as finish=<reason>
  - raise the default max_tokens to 8192
  - keep the needle count as the metric, always reported, never dropped
  - mark a truncated row FAILURE and annotate the metric, so a measurement
    artifact cannot enter the results as if it were a model result

9/10 stays SUCCESS: the README documents ~9/10 at >=20K as a known RDMA
residual, so a strict 10/10 gate would flag known-acceptable behaviour.
Logs predating the finish= field still parse.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 239755 (Kimi-K3, 2x8 on oci-64) died in vLLM's startup barrier. Node 0
finished loading weights at 23:21:41 and entered in_the_same_node_as ->
torch.distributed.barrier; node 1 was still constructing MoE layers ~50 min
later. gloo gave up at 23:51:41, exactly 1800s in, and the master's TCPStore
died with it, so node 1 then failed with "Broken pipe" to :29500. The barrier
is all-or-nothing across all 16 ranks, so what has to stay small is the GAP
between nodes, not the absolute load time.

Five problems, addressed here:

1. --distributed-timeout-seconds does not reach the barrier that failed. It
   feeds ParallelConfig.distributed_timeout_seconds, which only configures the
   device/NCCL groups. The startup barrier runs on the gloo CPU group, built by
   GroupCoordinator from the separate cpu_distributed_timeout_seconds field.
   Unset, it falls back to PyTorch's stock 1800s -- which is exactly the
   deadline in the traceback, despite 7200 being passed. Pass
   --cpu-distributed-timeout-seconds too.

2. Cold AITER JIT compile on every rank, every run. The image points
   AITER_JIT_DIR/TRITON_CACHE_DIR/VLLM_CACHE_ROOT/COMGR_CACHE_DIR at
   /opt/vllm_cache, but this launcher mounted only /tmp/vllm_cache, a path
   nothing reads, inside --rm containers. Mount a host-persistent cache at
   /opt/vllm_cache keyed by image ID, as vllm_dissag/run_xPyD_models.slurm
   already does. Per-node cold compile is precisely the skew the barrier
   cannot absorb.

3. WORKER_PID was tee's PID. After `a | b &` the shell reports b, so every
   kill in this script hit tee and left vLLM running, and no liveness check
   was possible. Launch through process substitution instead.

4. A dead engine was indistinguishable from a slow one. The head polled the
   log for the full 4000s after its writer had already died, then reported a
   timeout rather than the traceback. Both wait loops now check liveness --
   via a helper that also rejects zombies, since an exited-but-unreaped child
   still answers kill -0. This also fixes a latent hang where a worker would
   spin forever on a dead engine.

5. Container env was a closed allowlist. Tuning any of the above from CI meant
   a new -e line and another MAD PR. COLOCATED_FORWARD_ENV names the variables
   to forward, and they are passed via docker --env-file so values containing
   spaces survive. The file is node-local, mode 600, and unlike the wrapper
   scripts is never archived as a build artifact.

Also adds an opt-out checkpoint page-cache pre-warm before the container
barrier (PREWARM_CHECKPOINT=0 to skip). Its per-node duration is the real
diagnostic: a large spread means the storage path is the problem and no
timeout will fix it.

Verified: bash -n clean on both files, and on the single-quoted srun body
extracted in isolation; env-file forwarding preserves spaces and skips unset
vars at mode 600; the liveness helper correctly reports an unreaped child as
dead; DRY_RUN=1 emits the new flag and honours an override, with
COLOCATED_EXTRA_ARGS still appended last.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 240624 spent its entire 7200s allocation in the pre-warm added by the
previous commit. Both nodes printed "[prewarm] reading ..." and neither ever
printed "[prewarm] done", so vllm serve was never launched -- the run timed
out having tested nothing.

The flaw: it reads the WHOLE checkpoint on EVERY node, while PP2xTP8 means a
node only ever loads its own shard. On a 1453 GiB checkpoint that is roughly
4x the necessary I/O against a single NFS export, with both nodes competing
for it. It manufactured exactly the contention it was meant to relieve.

Default it to off. It stays available, because the per-node duration is a
useful storage probe -- a large spread between nodes means the storage path is
the problem and no timeout will fix it -- but it is now opt-in via
PREWARM_CHECKPOINT=1 and bounded by PREWARM_TIMEOUT_SECONDS (default 900). On
expiry it warns that the filesystem could not deliver the checkpoint in the
window and proceeds, rather than burning the whole allocation.

Note this run did not exercise --cpu-distributed-timeout-seconds: vLLM never
reached the startup barrier, so the absence of the 1800000ms gloo timeout in
the log says nothing either way. That fix is still untested.

Verified: default unset is silent; PREWARM_CHECKPOINT=1 runs and reports its
duration; a stalled read hits the budget, warns, and exits 0 so startup
continues.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mori_ep_env.sh:72-73 sets

    export NCCL_DEBUG="${NCCL_DEBUG:-INFO}"
    export NCCL_DEBUG_SUBSYS="${NCCL_DEBUG_SUBSYS:-INIT,NET,GRAPH}"

The ${VAR:-default} idiom advertises that a caller can override these. No
caller can. run_xPyD_models.slurm hands the container a fixed 22-name -e
list, and NCCL_DEBUG is not on it, so whatever the submitter exports is
dropped at the container boundary and the default always wins. The override
contract in mori_ep_env.sh is dead code as far as anything outside the
container is concerned.

Two things follow from that, and neither is hypothetical -- both come from
job 246618, a Qwen3-32B 1P1D run that completed successfully:

1. Log volume. That run archived 19 MB of per-rank logs (decode_NODE1.log
   9.6M, prefill_NODE0.log 4.4M, proxy_NODE0.log 2.7M), overwhelmingly NCCL
   channel-tuning triples -- pre-adjustment / minNChannels / post-adjustment
   -- emitted by all 8 ranks for every collective size.

2. Corrupted lines. Those 8 ranks share one stdout with no line atomicity,
   so at INFO volume the writes interleave mid-token:

       useocp[2026-09-07 16:09:52 TP0] Prefill batch, ...
       justment based on threadThreshold
       80 nc:112

   Any grep-based marker extraction over these logs is unreliable, which
   matters because CI reads them to decide pass/fail.

Narrowing NCCL_DEBUG_SUBSYS is not the fix. This run already had the
narrowed value INIT,NET,GRAPH and still produced all of the above; the
channel-tuning lines are GRAPH. Only NCCL_DEBUG=WARN suppresses them.

Forward the three NCCL debug knobs conditionally, using the same
${VAR:+-e VAR=$VAR} idiom vllm_dissag/run_xPyD_models.slurm already uses for
AITER_JIT_DIR and VLLM_ROCM_USE_AITER_MLA. When the variable is unset the
expansion is empty and the docker run is byte-identical to before, so the
default INFO behaviour is unchanged for anyone who does not opt out. Add the
same treatment for any further knob by adding one line.

The submitter's environment reaches the compute node because sbatch defaults
to --export=ALL and srun propagates it, so an `export NCCL_DEBUG=WARN` before
sbatch is now enough.

Deliberately not changing the INFO default here. INFO is a poor default for a
benchmark harness that archives its logs, but that is a call for the people
who use this interactively; this commit only makes the documented override
work.
models.json forced USE_CX7_NICS=0 on every sglang_disagg entry, which sends
prefill->decode KV transfer over mlx5_1, the 100G management NIC, instead of
the 8x 400G CX7 rail NICs that mori_ep_env.sh defaults to. That is the
critical path of a disaggregated run, so the measured throughput does not
represent the hardware. Two runs on different node pairs (AMD-internal
247073 and 247580) both show --disaggregation-ib-device mlx5_1.

Flipped to 1 on the four models this evidence covers: llama-3.1-8b,
qwen3-32b, llama-3.3-70b-fp8 and llama-3.1-405b-fp8. The remaining twelve
entries keep 0 until someone benchmarks them.

The rail NICs require same-rail nodes, and the file header has been citing
select_same_rail_nodes() as the thing that enforces that -- a function which
was never written. Written now: it maps each allocated node to its Level=0
switch via scontrol show topology and aborts before any container starts if
they do not share one. A cross-rail pair hangs in NCCL ibv_modify_qp rather
than failing, and base_flags sets --watchdog-timeout 1000000, so without
this the job burns its whole allocation instead of failing in seconds.

The check is skipped entirely when USE_CX7_NICS != 1, degrades to a warning
when no topology is configured, and can be bypassed with REQUIRE_SAME_RAIL=0
-- topology.conf may describe the management network rather than the CX7
fabric, in which case the check is conservative rather than correct.

--switches=2 is deliberately unchanged: both observed allocations were
already single-leaf, so constraining placement is the one change here that
could move the numbers.
decode.tp asked for --cuda-graph-bs $(seq 1 512), so 512 CUDA graphs were
captured at startup while benchmark_xPyD.sh only ever sweeps
CON="8 16 32 64 128 256 512". Startup was 51-52% of wall clock across two
runs (426s of 810s, then 401s of 784s).

Set the list to those seven values on the three models that used the full
range. Llama-3.1-405B keeps $(seq 1 8): it is already small and raising it
has memory implications this evidence does not cover.
py-spy, flask and pyyaml were installed from PyPI on every node at the start
of every run, so the pinned image was mutated before each benchmark: two
runs of the same tag are not the same environment, resolution can pick a
different version than the one requested, and a PyPI outage fails the run.
Observed in AMD-internal 247073/247580, where pip resolved click 8.5.0 and
installed 8.4.1, and the force-reinstall surfaced an unrelated xgrammar /
apache-tvm-ffi conflict in the log on both nodes.

Baked into docker/sglang_disagg_inference.ubuntu.amd.Dockerfile. The
launcher scripts now install only what is genuinely missing, so a MAD-built
image does no work there while an image supplied through DOCKER_IMAGE_NAME
still self-heals. --ignore-installed --force-reinstall is dropped.
Brings the branch onto current develop (652bad5) so the sglang_disagg fixes
and the kimi-k3 work can ship in one PR. Eight files conflicted:

models.json
  develop's 61b5301 migrated the registry from the root file to per-directory
  models.json. Took develop's empty root and moved this branch's four entries to
  where they now belong: pyt_vllm_disagg_mori_kimi-k3 into
  scripts/vllm_dissag/models.json, and the three pyt_vllm_kimi-k3_mi300x_*
  entries into a new scripts/vllm_multinode/models.json. The 26 jax_maxtext and
  primus_train entries in the old root file were NOT migrated: develop generates
  those from scripts/{jax-maxtext,primus_train}/get_models_json.py.

scripts/sglang_disagg/run_xPyD_models.slurm
  Union of develop's 25 conditional -e forwards and this branch's three NCCL_DEBUG
  forwards (28 total). The CX7 flip that was applied to the old root models.json is
  re-applied to scripts/sglang_disagg/models.json here.

scripts/vllm_dissag/models.yaml
  Append/append. Kept both recipe blocks (develop's GLM-5.1-FP8, this branch's Kimi-K3).

scripts/vllm_dissag/connectors/moriio.sh
  Took develop's argument order (model_args last) from a33b622/b33923b, so recipe
  flags win over connector defaults. Compatible with the Kimi-K3 recipe.

scripts/vllm_dissag/benchmark_niah.py and .sh
  The two sides added orthogonal features: develop added multi-seed sweeps with
  mean/min/max plus a warmup pass, this branch added finish_reason so a truncated
  answer is distinguishable from a retrieval miss. Combined both: _request now
  returns (message, finish, error) and the cross-seed summary flags TRUNCATED.
  Kept develop's NIAH_MAXTOK=2048 default -- the truncation detector makes the
  smaller default safe, and NIAH_MAXTOK is -e forwarded for models that need more.

scripts/vllm_dissag/parse_to_csv.py
  The auto-merge left two parse_niah_log and two save_niah_perf_csv definitions;
  the later pair (this branch's found= parser) silently shadowed develop's. Since
  the merged summary emits develop's mean=/min=/max= format, dropped this branch's
  pair and kept develop's parser. Kept the --narrow flag.

scripts/vllm_dissag/run_xPyD_models.slurm
  Six hunks, all unions: MORI_EP_VALID_MODELS and WIDE_EP_ONLY_MODELS gain both
  GLM-5.1-FP8 and Kimi-K3; the BENCHMARK_SCRIPT case gains both agentic and niah;
  the -e list gains develop's NIAH_SEEDS/NIAH_WARMUP/SHAPE_WARMUP/
  USE_INDUCTOR_GRAPH_PARTITION alongside this branch's NIAH_* forwards.

Verified: no conflict markers; bash -n on all eight touched shell/slurm files;
py_compile on both Python files; json.load on all 21 registries and yaml.safe_load
on both models.yaml; benchmark_niah.py run against a stub server on both the
scoring and truncation paths and its log fed through parse_to_csv.py --niah to a
valid perf.csv; and the sglang rail guard replayed against the real job-247580
topology across all six cases.
…d step

Loading Qwen3-32B off /shared_inference measured 683s in job 249926 -- 67% of a
1015s bring-up, ~92 MB/s per node with both nodes pulling the same 63 GB through
the same NFS mount. scripts/common/preflight_weights.sh:2-5 records what the other
path is: /mnt/m2m_nobackup/models_blog is per-node local NVMe, non-uniform, while
/shared_inference is the uniform NFS share.

An earlier commit on this branch reordered the sglang launcher to probe NFS first,
to avoid a wasted srun round trip and the "<jobid>.N bash FAILED 1:0" row a missed
probe leaves in sacct. That trade is backwards: it spends 683s of NFS reads to save
a few seconds of probing, and it means a node that does have the weights locally can
never use them. Restored local-NVMe-first.

The sacct noise that motivated the reorder is fixed properly instead. check_model_path
now runs a probe that always exits 0 and decides from its output, so a miss in a
fallback chain is not recorded as a step failure. Two latent bugs fall out of the
same rewrite -- the old version read srun's exit code alone, so an srun that exited 0
while reporting from fewer nodes than were allocated, or one that exited 0 with no
output at all, both returned "available on ALL nodes" and sent the job on to load
weights from a path some rank cannot see. The new version requires one Found line per
allocated node.

Applied to both scripts/sglang_disagg and scripts/vllm_dissag; only sglang needed the
ordering change, vllm_dissag was already local-NVMe-first.

Verified by extracting the function and replaying it against a stubbed srun across
five cases (all-found, one-node-miss, srun-fails, short output, silent exit-0). New:
0,1,1,1,1. Old, same stub: 0,-,1,0,0 -- it accepts short and silent.

Note this does not by itself make Qwen3-32B load fast: jobs 247073, 247580 and 249926
all fell through to NFS, so that model is not currently staged on the local NVMe of
the nodes CI draws from. This makes the fast path reachable and cheap to probe; the
staging itself is a cluster-side action.
The merge moved four entries out of the root models.json into the per-directory
registries develop's 61b5301 introduced, but kept their root-relative paths. All
twenty pre-existing per-directory registries resolve dockerfile as ../../docker/X
and scripts as a bare filename, so the migrated entries pointed at
scripts/vllm_multinode/scripts/vllm_multinode/run_multinode.slurm and would not
have resolved at all.

All 117 entries across the 21 registries now resolve from their own directory.
…t Kimi-K3 offline

The merge brought develop's multi-seed NIAH support into benchmark_niah.sh, which
now reads NIAH_SEEDS and NIAH_WARMUP alongside NIAH_WORDS/MAXTOK/TIMEOUT.
scripts/vllm_dissag forwards all of them; scripts/vllm_multinode forwarded only
three, so a colocated niah run silently fell back to the defaults for seeds and
warmup no matter what was set at submit time. Also forwards SHAPE_WARMUP, which
benchmark_xPyD.sh reads and which is reachable from the colocated path via
BENCHMARK_SCRIPT=sweep. Both launchers now forward every NIAH_* the benchmark reads.

Kimi-K3 is the model this branch adds and the only cell argv_assert.sh did not
cover -- it tested moriio+TP on Llama and moriio+wideEP on DeepSeek-V3. Added a
Kimi-K3 section over both roles: the int4_per_group_32 requant (without which
gfx942 cannot codegen the a16w4 SiTUv2 kernel and vLLM dies with "LLVM ERROR: Do
not know how to expand this operator's operand!" in determine_available_memory,
per benchmark/kimi_k3/mi300x/README.md:160), exactly one --compilation-config per
role, the kimi_k3 reasoning parser, the prefill/decode split across cudagraph_mode
(NONE vs PIECEWISE), all2all backend (mori_high_throughput vs mori_low_latency) and
kv_role, and the two model-gate arrays.

The gate assertions are scoped to the arrays themselves rather than grepped over
the whole file: "Kimi-K3" appears in both MORI_EP_VALID_MODELS and
WIDE_EP_ONLY_MODELS, so a file-wide grep passed even with the model removed from
one of them. MORI_EP_VALID_MODELS is line-continued, so it is extracted as a block.

49 passed, 0 failed. Each new assertion group was mutation-checked against the
merged tree: removing --quantization-config from the Kimi-K3 recipe fails 3,
swapping the decode all2all backend fails 2, dropping Kimi-K3 from either gate
array fails 1 each. The first version of the gate assertion did not fail under its
mutation and was rewritten.
benchmark_xPyD.sh looped over "{1..$BENCHMARK_ITR}". Brace expansion runs before
parameter expansion, so that is the literal 6-character string, and the loop body
executed exactly once no matter what the variable held. Job 252775 ran with
BENCHMARK_ITR=2, printed "iter: {1..2}" into its own benchmark log, and swept each
concurrency once. scripts/vllm_dissag/benchmark_xPyD.sh already used seq.

The fix uncovers a second bug that the first one had been hiding. parse_to_csv.py
starts at "RUNNING: the benchserving script for iter: 1" and pairs each result block
with the RUNNING line in the section before it -- but an `if i > 1` guard skipped the
very first pairing. That was invisible while the marker never appeared: the search
failed, parsing began at byte 0, and the skipped section was the pre-warmup preamble,
which carries no RUNNING line anyway. Restore the marker and the skipped section
becomes the first sweep cell. Replaying job 252775's log with the loop fixed and the
parser untouched gives 6 rows instead of 7, silently missing con=8.

So: seq for the loop, drop the guard, and accept an optional prompt count in the
RUNNING regex so archived logs in the older format still parse. Replayed against
job 252775's benchmark log, the fixed parser reproduces the CSV that build 44
published, byte for byte, from a log in either format.

Fixing the loop alone would also have doubled every sglang_disagg sweep, since
run_xPyD_models.slurm defaulted the knob to 2. That default was only ever decorative,
so it moves to 1: observed behaviour is unchanged and the knob now works.
These comments cite the job IDs and measurements that motivated each fix. On a
long-lived branch those numbers are unresolvable -- a reader six months from now
cannot look up job 252775 -- while the mechanism they illustrate is permanent.
Keep the mechanism, drop the identifiers.

Comment-only: no changed line is code.
run_multinode.slurm runs under `set -euo pipefail` and expands
COLOCATED_FORWARD_ENV at line 153 and again inside the srun body. Nothing
assigns it - not the pipeline, not any model card - and `export NAME` on an
unset variable marks the name for export without giving it a value, so it stays
unset. Under `set -u` that expansion is fatal.

It is an optional knob, so unset is the normal case, and the launcher aborted on
the normal case: SLURM job 354061 printed its banner and died with
"COLOCATED_FORWARD_ENV: unbound variable" before touching a model. Every
vllm_multinode run has been failing this way, which is every Kimi card.

Introduced in db276a6, which added the forwarding mechanism and both use sites
but no default.

Reproduced under `set -euo pipefail`: unset -> exit 1 with that exact message;
with the default -> exit 0 and an empty forward list, and a set value still
forwards, single and comma-separated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
run_multinode.slurm hardcoded MODEL_DIR=/shared_inference/models_blog, so a
colocated run always pulled its checkpoint over NFS -- even on nodes that already
held a local NVMe copy. It is the only one of the three launchers that did this;
sglang_disagg/run_xPyD_models.slurm and vllm_dissag/run_xPyD_models.slurm have
both probed /mnt/m2m_nobackup first for a while.

It also never bind-mounted /mnt/m2m_nobackup into the container, so pointing
MODEL_PATH there by hand could not have worked either.

This matters most for exactly the models people complain about. A cluster survey
found Kimi-K3 staged on local NVMe on all ten nodes that answered, at full size,
while every colocated run read it over NFS. Small models hide the difference
because they fit in page cache; a 1.5 TB checkpoint cannot.

The probe requires the path on EVERY allocated node -- ranks share one MODEL_PATH,
so a partial hit is useless -- and falls back to NFS otherwise. An explicit
MODEL_DIR or MODEL_PATH still wins. The srun body always exits 0 so a miss does
not leave a FAILED step in sacct, and the mount is guarded on -d because docker
auto-creates a directory at a missing bind source, which would shadow the model
with an empty one.

Verified against the real function with per-node fake roots: NVMe on both nodes
-> NVMe; on one node only -> NFS; on neither -> NFS; explicit MODEL_DIR and
MODEL_PATH both bypass the probe; srun failing -> NFS. bash -n clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
We are about to run the same cards two ways -- STANDALONE (the pipeline sbatches
the card's script) and MADENGINE (madengine generates the sbatch). Today the two
agree only by coincidence, because each launcher hardcodes the site's facts
separately. That drift has already cost real runs: run_multinode.slurm hardcoded
the NFS model dir and never probed local NVMe, so every colocated run read its
checkpoint over NFS while a full local copy sat on the node. Both siblings had
probed NVMe for a while; counting m2m_nobackup across the three gave 5/6/1, and
the odd one out was the bug.

scripts/common/cluster.sh follows the pattern vLLM uses at
.buildkite/amd-disagg/cluster.sh: sourced, every value defaulted with :-, so the
layering is always "environment beats site default". That single property is what
makes the two paths interchangeable, because both speak only environment
variables -- STANDALONE through standalone_env.sh and sbatch --export=ALL,
MADENGINE through deployment/slurm.py merging the card's env_vars into the sbatch
it writes. cluster.sh only fills what neither set, so neither path changes.

It also absorbs the two byte-identical copies of check_model_path, and drops a
default of SLURM_SUBMIT_DIR that named one person's home directory.

Card schema: the node count lived in one of two mutually exclusive places -- 39
cards had it only in args (the -N flag), which madengine never parses, and 4 had
distributed.nnodes + slurm.nodes with args empty. madengine reads
distributed.nnodes (deployment/slurm.py:677), so on that path the 39 fell back to
a default and pyt_large_ep_bench_1n/_2n/_4n would all have run on the same node
count. nnodes is DERIVED from the -N already in args, never invented; args is
untouched, and the pipeline only consults nnodes when args is blank, so
STANDALONE is unaffected.

Verified locally, no cluster needed:
- both-paths-agree: for all 43 multinode cards, the env STANDALONE produces and
  the env MADENGINE produces resolve identically through cluster.sh -- with no
  overrides, with image/model overrides, and with site overrides.
- resolver replayed against per-node fake roots under both shell regimes
  (set -euo pipefail, and no flags at all): NVMe on all nodes picks NVMe; on a
  subset falls back to NFS; on neither is fatal; an empty staged directory loses
  to a working NFS copy; an explicit MODEL_PATH bypasses the probe; a distinct
  MODEL_DIR is honoured last; srun failing moves to the next candidate.
- orphan-variable scan across all three launchers is clean, and re-introducing
  the COLOCATED_FORWARD_ENV bug makes it fail, so it would have caught d9a22b7.
- every name a launcher stopped defining is provided by cluster.sh; the six that
  are not are function-local scratch from the deleted duplicates, with zero
  remaining references.
- the -N in args equals distributed.nnodes equals slurm.nodes for all 43 cards;
  every models.json round-trips byte-exactly, so the diff is additive only.
- bash -n on all four shell/slurm files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
benchmark_xPyD.sh sweeps CON="8 16 32 64 128 256 512" -- seven concurrencies.
Mixtral-8x7B, DeepSeek-V3 and DeepSeek-R1 still declared
decode.tp: --cuda-graph-bs $(seq 1 512), so the server captured 512 CUDA graphs
at startup and used seven of them.

Measured on a 2-node DeepSeek-V3 run in tp mode (DP_MODE=0, so decode.tp is the
line in play), from the decode node's own log:

    Init torch distributed ends     30 s     5%
    Load weight end                 83 s    13%
    Memory pool end                 22 s     3%
    Capture cuda graph end         512 s    79%   <-- 511.58 s, 512 sizes
    ----------------------------------------
    total to ready                 651 s

Capture was four fifths of bring-up. The three sibling models at the top of this
file (Llama-3.1-8B, Qwen3-32B, Llama-3.3-70B-FP8) already ship exactly this seven
-value list in the same slot, and the decode.dp variants already use a short
explicit list too, so this makes the file internally consistent rather than
introducing a new convention.

Llama-3.1-405B-Instruct-FP8-KV keeps $(seq 1 8) deliberately: it is already small
and raising it has memory implications on a 405B model that this evidence does
not cover.

Safe for batch sizes that are not captured: sglang pads a running batch up to the
nearest captured size, which is why the seven-value list already works for the
three models using it.

Verified: every decode.tp capture list now equals CON exactly except the 405B
entry; models.yaml still parses; no $(seq 1 512) remains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cemberk and others added 24 commits November 16, 2026 06:21
Byte counts tuned for one GPU (a 40 GB KV budget on 288 GB) are wrong on
another. A recipe can now say

    memory:
      KV_CACHE_MEMORY_BYTES: {fraction: 0.104}   # of one GPU's memory
      MORI_SHMEM_HEAP_SIZE: {gib: 16}

and an env value of the same name still wins, so today's validated byte
counts stay as explicit overrides.

- models.py resolves memory: entries into env (with_memory); a recipe
  with a fraction needs resolving like an alias, so the job's recipe
  file carries the numbers to the containers.
- cluster_gpu_memory_for_recipe: for such a recipe only, one GPU's
  memory -- MAD_GPU_MEMORY_BYTES, else the largest memory bank of the
  first GPU in the KFD topology, smallest across nodes; refused when
  unreadable. Called before config_apply_recipe in the vLLM launchers.
- Tests: units for the sizes and the CLI, the probe on a fake KFD tree,
  and DeepSeek-V3's 20 GB KV budget written as a tenth of a 200 GB GPU
  giving the identical engine command.

Snapshots unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compat/rocm.yaml now describes the node and the network too, and the
scripts read it instead of holding the facts:

- platforms (mi300x-oam-8, mi325x-oam-8, mi355x-ubb-8,
  mi455x-helios-tray) with default_platform; fabrics (cx7, ainic,
  thor2: identification, GID index, interface, drivers, KV NIC, rails,
  own settings) with fabric_detection; scale_up_fabrics (xgmi, ualoe).
  recipe.py validate checks their references.
- cluster.sh: the fabric family and its fallbacks come from
  scripts/common/platform/fabric.py (the shell detection and the
  per-family case blocks are gone; a new NIC family is a catalog entry);
  GPUS_PER_NODE from the run's platform (MAD_PLATFORM, else
  default_platform). The GPU probe now also counts each node's GPUs, and
  cluster_verify_platform refuses a named platform whose accelerator the
  nodes do not have, or a node with fewer GPUs than the run uses.
- No script assumes 8 GPUs per node: the containers require
  GPUS_PER_NODE (the launchers pass it; SGLang now always forwards it),
  topology.py refuses an unknown count, reports record it as unknown.
- No script pins an arch: the connector .env MORI_GPU_ARCHS=gfx942 is
  gone (the image builds for its arch, the launcher passes the detected
  GPU), check_gpu_arch_declarations takes the known archs from the
  catalog, reports record an unknown arch as unknown, and the launchers
  forward a derived fabric by CLUSTER_FABRIC_DERIVED, not by family name.
- Tests: fabric families resolve exactly as before plus a catalog-only
  family and the family settings; platform_assert (platform sizes,
  verification); test fixtures that copy scripts/ also copy compat/.

Goldens: SGLang cards gain -e GPUS_PER_NODE=8; vLLM cards lose the
duplicate MORI_GPU_ARCHS=gfx942 the .env added before the detected value
(every rank's effective value is unchanged); one colocated card passes
MORI_GPU_ARCHS later in the list. Engine commands unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The connectors and the SGLang MoRI env fell back to one CX7 site's
values: rails mlx5_0,2,3,4,5,7,8,9, management NIC mlx5_1, GID index 3,
interface eth0, MoRI QoS TC 41 / SL 0. On another fabric those name
devices that do not exist, and NCCL quietly falls back to TCP.

- compat/rocm.yaml: cx7 gets those rails (connectors_choose: cluster.sh
  still leaves NCCL_IB_HCA to the connectors, as validated) and every
  family its rdma_qos.
- cluster.sh exports the detected family's values as MAD_FABRIC_RAILS,
  _GID, _IFACE, _KV_NIC, _RDMA_TC, _RDMA_SL; node_mad_env_args (was
  node_role_env_args) forwards them by name with the recipe's values.
- moriio.sh, rixl.sh, mori_ep_env.sh, sglang_disagg_mori_io_ep.sh and
  node.sh use them where the literals were, and refuse to start without
  a fabric rather than guess one. fabric.py container-env prints them for
  a hand run or a test; the tests that start role scripts directly pass
  the CX7 family's.

Goldens: every card's docker run gains the six -e MAD_FABRIC_* names
(values equal the former literals on the test fabric); engine commands
unchanged. SLURM and Kubernetes path equivalence unchanged (54/0, 54x2/0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compat/rocm.yaml `clusters` holds example site profiles (platform, fabric
family, site defaults such as storage roots); a run names one with
MAD_CLUSTER, or a site keeps its own profile file outside this public
repository and passes MAD_CLUSTER_FILE. cluster.sh applies it first,
before its own fallbacks, and only where the run passed nothing; an
unknown site, platform or fabric is refused. recipe.py validate checks
default_platform and the cluster entries.

Tests: platform_assert covers a catalog example, a site file, precedence
and refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A topology spec can now say what it runs on and how it is placed; a spec
using none of this resolves exactly as before (spec-file equivalence
unchanged, and a test that v1 specs gain no new per-node values).

- platform (GPUs per node and the scale-up domain from compat/rocm.yaml),
  partition (logical GPUs: compute-partition sizes are in the catalog for
  gfx942/gfx950; CDNA 5's are not stated and are refused), roles sized in
  gpus, a role on another platform, role kind (draft, encoder, embedding,
  kv_cache besides today's), kv_format agreed at the prefill-to-decode
  boundary, a replicas parallel mode.
- placement: same_domain / spread_domains, checked against the rank
  order; check() notes what crosses scale-out (an EP all-to-all larger
  than the domain) and, under strict, refuses a TP group larger than the
  domain or an EP group that would fit one but straddles two. Domains
  follow each role's own platform and count logical GPUs.
- Each node gets MAD_TOPO_DOMAIN (v2 specs only); in the per-task
  dispatch modes the leader names a failing rank and its domain.
- scheduler_hints: --segment=<nodes per domain> for SLURM's block
  topology, pod affinity on the site's domain label for Kubernetes.
- Launchers refuse what they cannot yet run (mixed node types, kinds
  other than prefill/decode/router/server).
- Examples in topology/specs/v2: Helios wide-EP 72 per rack, a TP-8
  server across two trays, MI355X CPX multi-tenant, heterogeneous
  prefill/decode, a speculative draft role; 17 new tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pins.py validate refuses a pin set whose architectures are not catalog
  accelerators or have no kernels entry for its build, so an image is
  never built that recipes cannot be checked against (when the catalog is
  at hand; a Docker build sees only docker/pins).
- recipe.py bringup --arch: the compiler target, the pin sets that build
  for the GPU and the components they pin there, the engine builds the
  catalog lists, and what is missing -- for CDNA 5 today: the target, a
  pin set, an engine build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- requires: {min_gpu_memory_gb: N}, checked against the run's platform
  product (MAD_PLATFORM), else the accelerator's largest product, so an
  unknown product never refuses a run.
- accuracy: records must say task, score and date (a variant's quality
  validation, for the best-known-configuration registry).
- variants under roles: may be keyed by a platform as well as an
  accelerator; the platform's applies over the accelerator's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- compat/rocm.yaml: peak dense TFLOPS per format for gfx942, gfx950 and
  CDNA 5, scale-up GB/s per GPU and NIC speeds per platform (vendor
  figures, to be calibrated).
- perf/model.py: prefill and decode tokens/s per GPU from those and a
  model's architecture (perf/models.yaml: DeepSeek-V3, Llama-3.1-405B,
  from their public configs); batch per GPU, MoE weights touched spread
  over the EP pool, the all-to-all on scale-up when the EP group fits a
  domain else on the NICs. `search` ranks platforms x pool sizes x batch
  x native formats and drops layouts whose weights do not fit.
  perf/calibration.yaml (measured / modelled per platform) corrects it
  once runs exist.
- perf/registry.py: best-known configurations as JSON lines keyed by
  model, platform, pin set, serving type, with SLO, accuracy and what
  reproduces the run; best() honours the SLO and an accuracy floor. Where
  the shared registry lives is still to be decided.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/common/serving_types.yaml lists each serving type from the plan
with its spec example, the launcher that runs it (or none yet), its
benchmark and its accuracy check (NIAH for long context; the rest are
still to build). New examples: a multimodal encoder role and an
embedding server. A test checks every named file exists, that the named
launcher accepts the example, and that every launcher refuses an example
marked as having none -- so the file cannot claim support that is not
there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The launchers know which recipe file, engine, roles and flags each of
them checks (cluster_check_recipe). precheck.py states that per launcher
directory and runs the same check from the submit host, so madengine
(or anything else that submits a card) can refuse a recipe that cannot
run on the target GPU before the job waits in a queue. The launcher still
checks on the allocated nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A table of what to change to add an accelerator, an engine build, a node
type, a NIC family, a site, a number format, a recipe for a new GPU and a
serving type, with the check that proves each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds on

A recipe written as variants (no alias pinning one) was applied with its
default_arch variant whatever the nodes were. The launchers now find the
GPU first for such a recipe (cluster_gpu_arch_for_recipe; any other
recipe probes nothing new, and the later arch gate reuses the result),
and config_apply_recipe, the job's recipe file and the recipe check all
take that GPU's variant. So the one Kimi-K3-MXFP4 card runs its gfx942
recipe on MI300X (int4 conversion) and its gfx950 recipe on MI355X
(native MXFP4) -- tested end to end; an alias still pins its own.

Snapshots unchanged (they set the GPU explicitly: no probe).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r runs

to_spec turns a search result into a v2 spec (prefill and decode pools of
its size on its platform, wide EP, each held inside one scale-up domain
when it fits); tested to validate and to be accepted by the vLLM disagg
launcher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What MAD derives -- the fabric family's MAD_FABRIC_*, CLUSTER_FABRIC_DERIVED,
the platform's accelerator, the GPU count probe, the recipe roles'
MAD_ROLE_ENGINE_ARGS_* / MAD_ROLE_ARTIFACT_* / MAD_ROLE_MODEL_PATH_*, the
job's recipe file -- was set unconditionally in places, so a value passed
through config (submit env, card or additional-context env_vars,
mad-config, a site profile) could be overwritten. Each is now a default
that a passed value wins over, like the rest of the configuration
layers; config.resolved.json records where each came from.
passthrough_assert.sh checks all of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/common/perf/best_known.jsonl holds the best measured
configuration per model, launcher, platform, card shape and (optionally)
pin set; its `apply` is a set of recipe keys. models.py for_job lays the
matching record over the recipe, so config_apply_recipe, the job's recipe
file (every container) and the recipe check all run it -- the same in
STANDALONE and through madengine, since both run the launcher. The
platform is MAD_PLATFORM, else one of the detected GPU and GPU count;
the launcher finds the GPU first when a record exists for the model.
Values passed through config still win; MAD_BEST_CONFIG=0 runs the
recipe as written; MAD_BEST_REGISTRY names another file; the record is
named in config.resolved.json (MAD_BEST_KNOWN). A record never changes a
run's shape.

registry.py: records carry launcher and shape; `promote` copies the best
run meeting its SLO into best_known.jsonl, replacing the previous record
for the same key. The file starts empty (no measured runs yet), so every
card runs as before (snapshots and path equivalence unchanged).

Tests: end to end -- the record applies, config wins over it, it can be
switched off, a record for another shape or platform does not apply;
promote replaces rather than appends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kimi-K3 becomes one recipe with gfx942 (the MI300X recipe, unchanged) and
gfx950 (the MI355X one) variants. Five new cards carry no GPU fact -- no
skip_gpu_arch, GPU_ARCHS, GPUs per node, EP_TP_SIZE or weight conversion
(tag any_gpu): pyt_vllm_kimi-k3_pp2xtp8, _wideep_allgather,
_wideep_moriep (colocated) and pyt_vllm_disagg_mori_kimi-k3_auto,
_kimi-k3-mxfp4_auto (disaggregated). The recipe's variant decides the
GPU-specific settings, the roles add the int4 conversion on gfx942, the
site's context gives GPUs per node. The GPU-named cards stay as they are.

Proven offline: on gfx942 each runs the same engine commands as its
MI300X-named twin (goldens), on gfx950 the Kimi-K3-MXFP4 one runs the
MI355X card's (end-to-end test), and the colocated one converts only on
gfx942. Existing Kimi-K3 cards gain only -e MAD_RECIPE_FILE.

- check_gpu_arch_declarations counts every variant of a recipe.
- Tests that build a topology from a card read the recipe's variant for
  the card's GPU, as the launcher does (snapshot.launch_env); a card may
  run on fewer GPUs than its recipe supports.
- MAD's Kubernetes submit takes GPUs per pod from the run's platform
  (MAD_PLATFORM, else default_platform) when the card states none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A role's speculative: {method, model, num_speculative_tokens} becomes the
engine's own arguments from compat/rocm.yaml engines -- one
--speculative-config JSON for vLLM, the algorithm / draft-model / steps
flags for SGLang -- added to that role (MAD_ROLE_ENGINE_ARGS_<ROLE>); a
method the engine does not document is refused. So the existing launchers
run speculative decoding (serving_types: speculative is runnable; a draft
model on GPUs of its own stays expressible-only). The flag comparison is
now recomputed per call (the no-build fallback added arguments twice).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
benchmark_accuracy.py asks task sets (JSONL: id, prompt, answer, match
exact|contains|regex; files in ACC_TASKS) through the endpoint's
OpenAI-compatible chat API at temperature 0 and prints one summary per
set; parse_to_csv.py --accuracy turns them into perf.csv rows (the
score; FAILURE only when requests failed). The vLLM disaggregated and
colocated launchers select it with BENCHMARK_SCRIPT=accuracy; a 20-item
arithmetic smoke set (written here) is the default. serving_types lists it
for the runnable vLLM types. Tested against a stub endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
REPLICAS=1 on the colocated launcher -- or a topology spec with parallel
mode replicas, which vllm_multinode now accepts -- runs GPUS_PER_NODE /
TP_SIZE independent servers on every node, each on its own GPUs
(HIP_VISIBLE_DEVICES) and port, and replica_router.py (round robin,
standard library) on SERVE_PORT on the head node, so the sweep, NIAH and
accuracy benchmarks measure the whole fleet unchanged. The TP*PP check
becomes 'TP divides a node' in this mode. Multi-tenant serving
(MI355X CPX: eight TP-8 servers per node) is now runnable.

Tests: the router's round robin and health; a node's replicas, devices
and ports from a dry run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g role

benchmark_embeddings.py measures an OpenAI-compatible /v1/embeddings per
concurrency (inputs/s, p50/p99) and fails requests that error or return
the wrong number of embeddings; parse_to_csv.py --embeddings writes the
rows; BENCHMARK_SCRIPT=embeddings selects it on the colocated launcher,
which now accepts a topology role of kind embedding. The server is an
embedding model through its flags (vLLM: --runner pooling), typically
replicas of TP 1. serving_types: embeddings runnable. Tested against a
stub endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
topology.py scheduler-hints for Kubernetes: with the site's domain label
(k8s.domain_label / MAD_K8S_DOMAIN_LABEL), a spec whose placement keeps
the run in one scale-up domain gets pod affinity on that label for the
whole Job (labelled job-name), added by MAD's submit here and by
madengine's Kubernetes path alike. One Indexed Job has one pod template,
so roles in different domains (spread_domains) or a same_domain run
larger than a domain are refused with that reason (a Job per role is not
supported yet).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A role's lora: {modules: {name: path}, max_loras, max_rank} becomes the
engine's arguments from compat/rocm.yaml engines (vLLM --enable-lora
--lora-modules ..., SGLang --lora-paths ...) and exports LORA_ADAPTERS
(config wins; forwarded to the containers). benchmark_lora.py spreads
chat requests over the adapters per concurrency (parse_to_csv.py --lora);
BENCHMARK_SCRIPT=lora on both vLLM launchers. serving_types: multi-LoRA
runnable. Tested against a stub endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/common/workloads: a library of workload layers (madengine layered
config v2, OmegaConf defaults) -- base, serving shapes (disagg wide EP,
colocated, replicas, SGLang DP+EP), benchmarks (sweep, NIAH, accuracy,
embeddings, LoRA), model families (Kimi-K3, DeepSeek-V3).

Eight example cards (tags e2e_example, any_gpu), each a few lines of
workload on those layers, with alias recipes that inherit validated ones:
Kimi-K3 disagg on any GPU with an example best-known record; DeepSeek-V3
MTP speculative decoding and KV sized from GPU memory; DeepSeek-V3
disaggregated quantization (per-role checkpoints, refused on gfx942);
Kimi-K3 on Helios (72-GPU pools per rack, refused until CDNA 5 bring-up);
Llama-3.1-405B colocated TP8xPP2; Qwen3-32B replicas with LoRA;
Qwen3-Embedding-8B replicas; DeepSeek-R1 on SGLang.
test_e2e_examples.py runs each through the real launchers on each GPU it
targets and asserts what it claims. docs/e2e-examples.md.

Fixes found on the way:
- The best-known message in config_apply_recipe used single quotes inside
  a single-quoted python3 -c program: whenever a record matched, the
  recipe layer was not applied. Rewritten; check_srun_quotes now checks
  python3 -c bodies too, where any apostrophe changes the program.
- A GPU a variant recipe has no variant for gets the default variant
  (whose GPU_ARCHS refuses it), never the bare base.
- The launchers' model allow-lists follow recipe aliases (models.py
  --base); MAD_BEST_REGISTRY relative paths are taken from scripts/;
  topology.py reports a missing catalog cleanly; from_env knows REPLICAS.
- The snapshot harness resolves workload files with madengine (so the
  way-4 twin card now runs, and runs exactly as its plain twin) and skips
  them, saying so, without madengine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The launchers apply the card's workload file (MAD_CONFIG, else a sibling mad-config.yaml, as
madengine locates it) with config/workload.py, MAD's own reader of madengine's layered config
(defaults: inheritance, ${...} and ${oc.env:...} interpolation, benchmark map by kind; standard
library and PyYAML). Values go in at submit level under anything already set, labelled "workload"
in config.resolved.json; a missing or unusable file stops the run.

- k8s/submit.py without madengine resolves the same layers instead of refusing the card.
- The snapshot harness and the e2e example tests no longer import madengine; when it is
  importable, tests check its reader gives the same environment (e2e cards, k8s pod env).
- config_layers_assert covers the workload layer; test_workload covers the reader.
- SGLang launcher: the MoRI decision runs after the workload layer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants