Deploy madengine workloads to Kubernetes, SLURM, or Spur, or run them on this machine.
madengine supports these deployment backends:
- Local Docker - Containers on this machine when no scheduler key is set
- Bare metal - A conda prefix on this machine. Selected with
"platform": "bare_metal", checked before every scheduler key. See bare-metal.md - Kubernetes - Cloud-native container orchestration
- SLURM - HPC cluster job scheduling
- Spur - Crusoe's SLURM-compatible scheduler. A
slurmblock withscheduler: spur, ordeploy: spur, selects it. See Spur - llm-d - Benchmarking a Kubernetes-native distributed inference stack; see llm-d.md
Deployment is configured via --additional-context and happens automatically during the run phase.
Build once, then deploy the resulting manifest to any target:
flowchart LR
B["1. Build Phase<br/>(local or CI/CD)<br/>madengine build --tags model"] --> M[(build_manifest.json<br/>image or conda prefix)]
M --> D["2. Deploy Phase<br/>madengine run --manifest-file build_manifest.json<br/>--additional-context '{...}'"]
D --> T{detect target}
T -->|platform bare_metal| B[conda prefix on this machine]
T -->|llm_d| I[llm-d benchmark Job]
T -->|k8s / kubernetes| K[K8s Job]
T -->|slurm scheduler spur| U[Spur job array]
T -->|slurm| S[SLURM script]
T -->|neither| L[Local Docker]
No explicit deploy field is required — the target is inferred from the config structure (Convention over Configuration):
flowchart TD
A[additional_context] --> P{platform bare_metal?}
P -->|yes| B[Bare metal conda]
P -->|no| Q{which key?}
Q -->|llm_d| I[llm-d benchmark]
Q -->|k8s / kubernetes| K[Kubernetes deployment]
Q -->|slurm scheduler spur, or deploy spur| U[Spur job array]
Q -->|slurm| S[SLURM deployment]
Q -->|neither| L[Local Docker execution]
platform: bare_metal is checked first. It cannot be combined with slurm, k8s, kubernetes, or llm_d. llm_d is checked next: an llm-d config also carries a k8s block, which configures the benchmark-client Job. A slurm block whose scheduler is spur, or a top-level deploy: spur next to a slurm block, selects Spur at build time and at run time. Omitted platform stays Docker.
Batch Jobs force the pod template label job-name to equal metadata.name, and
label values are at most 63 characters. madengine therefore builds Job names with
sanitize_k8s_object_name(..., max_total_len=63) so long model card paths (for
example llm-d MoE cards) hash/truncate instead of failing admission. PVC and
ConfigMap names append a short suffix (-results, -config) and may still use
the longer RFC 1123 subdomain limit (253).
- Kubernetes cluster with GPU support
- GPU device plugin installed (AMD or NVIDIA)
- Kubeconfig configured. madengine uses
k8s.kubeconfig, else$KUBECONFIG, else~/.kube/config(in-cluster when none exists), atk8s.contextor that file's current context. Every API call and everykubectlcall it makes uses that same file and context, and ak8s.contextmissing from the file is refused rather than substituted. A rerun replaces only objects labelledapp=madengine; a same-named object without the label stops the deploy. - Docker registry accessible from cluster
{
"k8s": {
"gpu_count": 1
}
}This automatically applies intelligent defaults for namespace, resources, image pull policy, etc.
# 1. Build image
madengine build --tags my_model \
--registry my-registry.io \
--additional-context-file k8s-config.json
# 2. Deploy to Kubernetes
madengine run \
--manifest-file build_manifest.json \
--timeout 3600The deployment target is automatically detected from the k8s key in the config.
k8s-config.json:
{
"k8s": {
"gpu_count": 2,
"namespace": "ml-team",
"gpu_vendor": "AMD",
"memory": "32Gi",
"cpu": "16",
"image_pull_policy": "Always"
}
}Configuration Priority (lowest to highest precedence):
- Base defaults
- GPU vendor preset (AMD/NVIDIA)
- GPU vendor multi-GPU preset (AMD only, for multi-GPU/multi-node)
- Profile preset (single-gpu/multi-gpu/multi-node)
- User config (
--additional-context-file) — highest precedence, applied last
See examples/k8s-configs/ for complete examples.
By default (k8s.secrets.strategy: from_local_credentials), madengine run creates Kubernetes Secrets from a local credential.json when present: Docker Hub pull credentials (when configured) and an opaque Secret for runtime use. Pods also reference the pre-created pull secret dockerhub-rocm (k8s.secrets.image_pull_secret_names). Credentials are not embedded in the ConfigMap in that case. For GitOps or clusters without client-side files, use existing or omit and set k8s.secrets.image_pull_secret_names / k8s.secrets.runtime_secret_name as needed. See Configuration and examples/k8s-configs/README.md.
With "debug": true in additional context, madengine run writes rendered manifests under ./k8s_manifests (or the path you configure). To lint those YAML files against the Kubernetes OpenAPI schema, install kubeconform and run from the repository root:
./tests/scripts/k8s_validate_manifests.sh ./k8s_manifestsThe script exits successfully if kubeconform is missing (skip) or if validation passes.
A multi-node job uses a shared ReadWriteMany volume when k8s.nfs_storage_class (default nfs-banff) can serve one. On a cluster whose only StorageClass is local-path, each pod gets its own ReadWriteOnce disk (results_layout / data_layout: auto, or force shared / per_pod). Performance CSVs are then taken from pod logs. See examples/k8s-configs/README.md.
madengine run --additional-context distributed (launcher, nnodes, nproc_per_node) is applied at submit time and wins over the copy saved in build_manifest.json. n_gpus: -1 on a model card means "all GPUs on the node", not a process count. Multi-node vLLM and SGLang pods join the headless Service subdomain so pod-0 DNS resolves.
For distributed training across multiple nodes:
{
"k8s": {
"gpu_count": 8
},
"distributed": {
"launcher": "torchrun",
"nnodes": 2,
"nproc_per_node": 4
}
}This creates:
- Kubernetes Indexed Job with 2 completions
- Headless service for pod discovery
- Automatic rank assignment via
JOB_COMPLETION_INDEX MAD_MULTI_NODE_RUNNERenvironment variable with torchrun command
Supported Launchers:
torchrun- PyTorch DDP/FSDPdeepspeed- ZeRO optimizationmegatron-lm- Megatron-LM trainingtorchtitan- LLM pre-trainingprimus- Primus unified pretrain (Megatron / TorchTitan / MaxText YAML)vllm- LLM inferencesglang- Structured generationsglang-disagg- Disaggregated SGLang (multi-node)slurm_multi/slurm-multi- Self-managed multi-container topologies (escape hatch; on Kubernetes see slurm_multi cards on Kubernetes)
See Launchers Guide for details.
A slurm_multi card ships its own SLURM batch script, which fans out with srun and
starts one container per node with docker run. On Kubernetes madengine runs that
script unchanged:
- one pod per node of the card: an Indexed Job (
completions = nnodes, one attempt) with a headless Service, so pods reach each other by hostname (<job>-0,<job>-1, ...); - every pod runs the card's image; pod 0 runs the card's script with the
SLURM_*variables a batch script sees, and with stand-ins forsrun,dockerandscontrolfirst onPATH.srunsends each task to the named pod;docker runruns its command in the pod itself (the pod is the container), applying-eand-v;docker stop/rm/ps/pull/image inspectact on those processes and the pod's image;scontrol show hostnamesexpands the node list. Any other form fails with a message naming it, rather than being guessed at; - the card's script directory and its sibling
commondirectory travel in a ConfigMap (compressed, under 1 MiB); - results:
LOG_PATHdefaults to/results/logson the run's results volume, which every pod mounts (ReadWriteMany for multi-node). Pod 0 publishes the card'smultiple_resultsfile, else$LOG_PATH/$SLURM_JOB_ID/perf.csv, as/results/<job>-0/perf.csv. madengine copies the run into./k8s_results/<job>/run-<id>/, reads exactly that file -- never an earlier run's -- and treats its rows the way the SLURM path does: each row'sstatusdecides success or failure, and no file is a failed run; - the run is done when the Job's own
CompleteorFailedcondition says so (not when the first pod finishes), and it has one attempt: rerunning one pod of a multi-node run is not a retry of the run; - the run's name is a DNS label with room for the pod index (dots become dashes, a long name is cut and given a short hash), because the script addresses nodes by hostname.
What the script needs from the cluster is pod-spec configuration, never a card change:
k8s key |
Purpose |
|---|---|
gpu_count |
GPUs per pod (default: the card's slurm.gpus_per_node). Presets do not size the pod: see below |
cpu, memory |
requests per pod; requested only when the run sets them |
volumes |
[{name, mount_path, host_path | pvc | nfs: {server, path}, read_only}] — weights, caches |
extra_resources |
other device-plugin resources, e.g. {"rdma/hca": 1} |
security_context |
container securityContext, e.g. {"capabilities": {"add": ["IPC_LOCK"]}} for RDMA |
shm_size |
/dev/shm size (default 64Gi) |
include_script_dirs |
directories shipped with the card's (default: sibling common) |
image_pull_policy |
default Always; IfNotPresent for images imported into the cluster |
host_network |
pods on the node's network, so RoCE uses the node's NICs; one host-network slurm_multi pod per node, across runs and namespaces (madengine's agent and the card's servers bind fixed ports) |
pod_annotations |
e.g. {"k8s.v1.cni.cncf.io/networks": "rdma-net"}: a Multus secondary network giving each pod its own RDMA interface |
results_access_mode |
results PVC access mode (default ReadWriteMany for multi-node). ReadWriteOnce works when every pod lands on one node, e.g. a cluster with only local-path storage |
Host-level docker run options in the script (--device, --network host,
--privileged, --ulimit, --shm-size, ...) are ignored; the keys above are where
the pod gets those capabilities.
RDMA for a card's KV transfer takes three things from the cluster: RDMA devices in
the pod (an RDMA device plugin; request its resource with extra_resources and add
IPC_LOCK with security_context -- not privileged, which also exposes every GPU
on the node and defeats the GPU device plugin's isolation), a network the RDMA
traffic can use (host_network, or a Multus/SR-IOV network via pod_annotations),
and GPU memory the NIC can register (a GPU peer-memory driver or dma-buf support on
the node, outside Kubernetes). Preset env vars (for example the AMD multi-GPU
preset's NCCL_IB_DISABLE=1) are not passed to the card, as on SLURM; set them
under env_vars to pass them.
Presets do not size a slurm_multi card's pods either. They pick a profile from the run
config alone, before the card is known, so a config that names no gpu_count and no
distributed block gets the single-GPU profile (1 GPU, 8 CPUs, 16Gi). For these cards
GPUs per pod come from k8s.gpu_count only when the run sets it, else the card, and
cpu / memory are requested only when the run sets them. To set the node count from
the run, give distributed: {"launcher": "slurm_multi", "nnodes": N}: the multi-node
preset defaults the launcher to torchrun, and a run-level nnodes is read only for
slurm_multi.
MAD has a standalone counterpart that needs no madengine,
scripts/common/k8s/submit.py, the Kubernetes counterpart of submitting the card's
script with sbatch. It reads the same k8s keys and, for the same card and config,
deploys the same Job, Service and ConfigMap (apart from names and the run id) with a
byte-identical pod-side shim. MAD's docs/multinode-kubernetes.md covers both paths.
# Check job status
kubectl get jobs -n your-namespace
# View pod logs
kubectl logs -f job/madengine-job-xxx -n your-namespace
# Check pod status
kubectl get pods -n your-namespaceFinished Jobs are not removed unless you set k8s.ttl_seconds_after_finished to a positive number of seconds; the Job manifest then includes ttlSecondsAfterFinished so the control plane can garbage-collect the Job after it finishes. The deploy step may still delete Secrets it created when cleaning up a failed or cancelled deploy—see runtime logs for details.
Manual cleanup:
kubectl delete job madengine-job-xxx -n your-namespace- Access to SLURM login node
- SLURM commands available (
sbatch,squeue,scontrol) - SLURM-compatible schedulers that implement part of the CLI work too. On Spur, for example,
SLURM_NODELISTis already a comma list andscontrol show hostnamesdoes not exist, so madengine expands node lists itself and usesscontrolonly for the compressed form;sacct -Xis rejected, so the final state is read without it; andsbatchis retried when the Raft control plane is electing a leader ("not the Raft leader"). Theslurm_multiwrapper also honoursslurm.accountandslurm.qos - Shared filesystem for MAD package and results
- Module system or container runtime (Singularity/Apptainer)
slurm-config.json:
{
"slurm": {
"partition": "gpu",
"gpus_per_node": 4,
"time": "02:00:00",
"account": "my_account"
}
}# 1. Build image (on build node or locally)
madengine build --tags my_model \
--registry my-registry.io \
--additional-context-file slurm-config.json
# 2. SSH to SLURM login node
ssh user@hpc-login.example.com
# 3. Deploy to SLURM
cd /shared/workspace
madengine run \
--manifest-file build_manifest.json \
--timeout 7200The deployment target is automatically detected from the slurm key in the config.
slurm-config.json:
{
"slurm": {
"partition": "gpu",
"account": "research_group",
"qos": "normal",
"gpus_per_node": 8,
"nodes": 1,
"time": "24:00:00"
}
}Common SLURM Options:
partition: SLURM partition nameaccount: Billing accountqos: Quality of Servicegpus_per_node: Number of GPUs per nodenodes: Number of nodes (for multi-node)nodelist: Comma-separated node names to run on (e.g."node01,node02"); when set, job runs only on these nodes and node health preflight is skippedreservation: SLURM reservation name; forwarded to srun health/cleanup commandstime: Wall time limit (HH:MM:SS)exclusive: Exclusive node access (default:true)scheduler: SLURM flavor -slurm(default) orspur(see below)rendezvous_timeout: spur only; seconds a node waits for rank 0 to publishMASTER_ADDR(default: 900)
See examples/slurm-configs/ for complete examples.
Spur exposes SLURM-compatible CLI shims but srun cannot fan tasks out across
nodes, so multi-node runs use a job array of single-node tasks instead: each
array task runs on one node, SLURM_ARRAY_TASK_ID is the node rank, and the
tasks self-form the cluster through a shared-filesystem rendezvous (rank 0
publishes its transport IP; the other ranks read it as MASTER_ADDR).
Select it with slurm.scheduler; everything else in the slurm block is
unchanged:
{
"slurm": {
"scheduler": "spur",
"partition": "gpu",
"nodes": 4,
"gpus_per_node": 8,
"time": "02:00:00"
}
}A job array carries no gang-scheduling guarantee, so tasks may start minutes
apart. A node that does not see rank 0's address within rendezvous_timeout
fails the run with a diagnostic rather than starting with an empty
MASTER_ADDR; raise the timeout if your queue wait is longer than that.
slurm.output_dir must be on a filesystem shared by every node - it holds the
rendezvous files.
The srun-based node health preflight (enable_node_check) is skipped on spur:
srun -w <node> does not run on the requested node there, and the --nodelist
it would pin conflicts with the job array, whose tasks each request one node.
On stock SLURM, SGLANG_NODE_IPS is the job nodelist expanded to addresses, with 127.* skipped (Ubuntu maps the hostname to 127.0.1.1 in /etc/hosts). On Spur each array task's nodelist contains only that task's node, so the same expansion is a single IP and the router starts without --decode.
Every Spur rank publishes ip_rank<N> in the rendezvous directory before rank 0 writes MASTER_ADDR. Spur waits up to rendezvous_timeout and concatenates those files in rank order into SGLANG_NODE_IPS. The address is taken the same way as MASTER_ADDR: NCCL_SOCKET_IFNAME when it is set (default ens3 in the task script), otherwise hostname -I. Stock SLURM still uses the nodelist.
For distributed training across SLURM nodes:
{
"slurm": {
"partition": "gpu",
"nodes": 4,
"gpus_per_node": 8,
"time": "48:00:00"
},
"distributed": {
"launcher": "torchrun",
"nnodes": 4,
"nproc_per_node": 8
}
}SLURM automatically provides:
- Node list via
$SLURM_JOB_NODELIST - Master address detection
- Network interface configuration
- Rank assignment via
$SLURM_PROCID
# Check job queue
squeue -u $USER
# Monitor job progress
squeue -j <job_id>
# View job details
scontrol show job <job_id>
# Check output logs
tail -f slurm-<job_id>.outFor workloads that use externally maintained Docker images (e.g. SGLang, vLLM releases):
# Skip Docker build, use a pre-built image
madengine build --tags model --use-image lmsysorg/sglang:latest
# Auto-detect image from model card's DOCKER_IMAGE_NAME
madengine build --tags model --use-image auto
# Build on a SLURM compute node and push to registry
madengine build --tags model --build-on-compute --registry docker.io/myorgThe manifest generated by --use-image merges the model card's distributed and slurm config into deployment_config, so the run phase auto-detects SLURM deployment without additional --additional-context.
The --build-on-compute job takes its SLURM settings from the model card's slurm section and --additional-context (slurm wins): partition (required), time, reservation, and the same account and qos as the run job, so a cluster whose default QoS cannot get a node accepts the build too. A build needs no GPUs, so the job requests none unless slurm.build_gpus_per_node is set, for schedulers that refuse a job without a GPU request:
{"slurm": {"partition": "gpu", "account": "my-account", "qos": "my-qos", "build_gpus_per_node": 8}}For workloads that orchestrate their own per-node Docker containers (e.g. SGLang Disaggregated proxy + prefill + decode topologies), use the slurm_multi launcher:
{
"distributed": {
"launcher": "slurm_multi"
},
"slurm": {
"partition": "gpu",
"nodes": 3,
"gpus_per_node": 8,
"reservation": "my-reservation"
}
}Unlike templated launchers, slurm_multi runs the model's .slurm script directly on baremetal. The script manages its own Docker containers via srun internally. See Launchers Guide — slurm_multi for details.
When madengine run detects an existing SLURM allocation (SLURM_JOB_ID is set, e.g. inside salloc), the slurm_multi launcher runs the generated wrapper script synchronously with bash instead of nesting another sbatch. Templated launchers do the same for multi-node runs: the job script runs in place and starts its tasks with srun on the first nodes hosts of the allocation (or on slurm.nodelist, when set). Each run gets its own paths, so several runs can share one allocation. Single-node runs still submit their own sbatch, because in place their script would run on the submitting host.
Those paths are job-scoped directories, not the SLURM submission directory. A single-node job uses $SLURM_TMPDIR/madengine_job_<job id> when that directory is writable, otherwise /tmp/madengine_job_<job id>. A multi-node task appends _node_<rank>, and an in-place run also appends a per-run suffix so two runs in one allocation do not share a directory. A multi-node task removes its workspace only after the task exits 0 and every artifact has been copied out; a failure, or a copy that did not finish, keeps the directory. A later job from the same user deletes madengine_job_* directories older than 7 days under that scratch root when squeue -u lists this user's jobs and the directory's job id is not among them. An unanswered squeue deletes nothing.
salloc --nodes=3 --gpus-per-node=8 --partition=gpu
madengine run --manifest-file build_manifest.json
# → Detects salloc, runs synchronously# Cancel job
scancel <job_id>
# Cancel all your jobs
scancel -u $USER| Feature | Kubernetes | SLURM | llm-d |
|---|---|---|---|
| Environment | Cloud, on-premise | HPC clusters | Kubernetes |
| Orchestration | Automatic | Job scheduler | Inference Gateway + helm |
| Dependencies | Python library (kubernetes) |
CLI commands only | kubernetes + helm (managed mode) |
| Multi-node Setup | Headless service + DNS | SLURM env vars | InferencePool + Endpoint Picker |
| Resource Management | Declarative (YAML) | Batch script | Helm charts |
| Best For | Cloud deployments, microservices | Academic HPC, supercomputers | Benchmarking disaggregated serving |
Bare metal is the local conda path and is documented in bare-metal.md. Spur reuses the SLURM column's job model with a job array and a shared-filesystem rendezvous; see Spur.
{
"k8s": {
"gpu_count": 1,
"namespace": "dev"
}
}{
"k8s": {
"gpu_count": 4,
"memory": "64Gi",
"cpu": "32"
},
"distributed": {
"launcher": "torchrun",
"nnodes": 1,
"nproc_per_node": 4
}
}{
"k8s": {
"gpu_count": 8,
"namespace": "ml-training"
},
"distributed": {
"launcher": "torchtitan",
"nnodes": 4,
"nproc_per_node": 8
}
}{
"slurm": {
"partition": "gpu",
"gpus_per_node": 8,
"time": "12:00:00"
}
}{
"slurm": {
"partition": "gpu",
"nodes": 8,
"gpus_per_node": 8,
"time": "72:00:00",
"account": "research_proj"
},
"distributed": {
"launcher": "deepspeed",
"nnodes": 8,
"nproc_per_node": 8
}
}Image Pull Failures:
# Check image exists
docker pull <registry>/<image>:<tag>
# Verify image pull secrets
kubectl get secrets -n your-namespace
# Check pod events
kubectl describe pod <pod-name> -n your-namespaceNode Reported as FAILED but Pod Succeeded:
In multi-node jobs, madengine may report a node as FAILED even though Kubernetes shows the pod as Succeeded. This occurs when the kubelet on the node becomes unreachable after the job completes, preventing madengine from collecting stdout logs (and therefore parsing performance metrics).
To verify:
# Check actual pod status — if Succeeded, the workload ran fine
kubectl describe pod <pod-name> | grep Status
# Check the node's kubelet health
kubectl get nodes
kubectl describe node <node-name> | grep -A5 ConditionsPVC artifacts are still collected in this scenario. Only the API-based pod log retrieval fails, which means performance metrics for that node will be missing from the results table.
slurm_multi cards:
| Symptom | Cause and fix |
|---|---|
job name ... does not make valid pod hostnames |
Not expected since names are shortened automatically; report it with the card name |
Error: required file missing: .../scripts/<dir>/... in pod 0's log |
The card's script reads a sibling directory that was not shipped. Add it to k8s.include_script_dirs |
RuntimeError: The memory capacity is unbalanced |
The pod sees GPUs that are not its own, usually security_context: {privileged: true}. Remove it |
Pods Pending with Insufficient amd.com/gpu |
nnodes x gpu_count exceeds the free GPUs, or host_network needs one node per pod |
Pods Pending with didn't match pod anti-affinity rules |
host_network: another host-network slurm_multi run holds those nodes; this run starts when they free up |
Results PVC Pending |
No storage class satisfies ReadWriteMany. Set multi_node_results_storage_class, or results_access_mode: ReadWriteOnce with every pod on one node |
[mad-k8s srun] unsupported option ... |
The card's script uses an srun / docker / scontrol form the stand-ins do not implement; the message names it |
Resource Issues:
# Check node resources
kubectl describe nodes | grep -A5 "Allocated resources"
# Check GPU availability
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.capacity.'amd\.com/gpu'Job Pending:
# Check reason
squeue -j <job_id> -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"
# Check partition status
sinfo -p gpuOut of Resources:
# Check available resources
sinfo -o "%P %.5a %.10l %.6D %.6t %N"
# Adjust resource requests in config- Use minimal configs with intelligent defaults
- Specify resource limits to prevent over-allocation
- Use appropriate namespaces for isolation
- Configure image pull policies based on registry location
- Monitor pod resource usage with
kubectl top
- Start with conservative time limits
- Use appropriate QoS for priority
- Monitor job efficiency with
seff <job_id> - Use shared filesystem for input/output
- Test with single node before scaling
- Launchers Guide - Distributed training and inference launchers
- Bare metal - Conda execution on this machine
- llm-d Guide - Benchmarking the llm-d distributed inference stack
- K8s Examples - Complete Kubernetes configurations
- SLURM Examples - Complete SLURM configurations
- llm-d Examples - Complete llm-d configurations
- Usage Guide - General usage instructions