Complete documentation for madengine - AI model automation and distributed benchmarking platform.
| Guide | Description |
|---|---|
| Installation | Complete installation instructions |
| Usage Guide | Commands, configuration, and examples (--skip-model-run) |
| Guide | Description |
|---|---|
| Configuration | JSON and YAML --config; includes run log error pattern scan |
| Batch Build | Selective builds with batch manifests |
| Deployment | Kubernetes, SLURM, and Spur deployment |
| Bare metal | Conda execution on this machine, with no Docker |
| llm-d | Benchmarking the llm-d distributed inference stack |
| Launchers | Multi-node training frameworks |
| Distributed Config | Where each kind of setting belongs: run, site, model, measurement |
| Guide | Description |
|---|---|
| Profiling | Performance analysis tools |
| Contributing | How to contribute to madengine |
| Guide | Description |
|---|---|
| CLI Reference | Complete command-line options and examples |
| Codebase Wiki | Single-page architecture and internals reference (source) |
The CLI drives orchestrators that discover and build models, then hand off to a local or distributed execution target, which runs the model under the appropriate launcher and emits performance data for reporting. (Same diagram as the main README.)
flowchart TB
subgraph CLI["CLI Layer — Typer + Rich"]
C1[discover]
C2[build]
C3[run]
C4[report]
C5[database]
C6[cleanup]
C7[setup-page]
end
subgraph ORC["Orchestration Layer"]
O1[DiscoverModels]
O2[BuildOrchestrator]
O3[RunOrchestrator]
MAN[(build_manifest.json)]
end
subgraph EXEC["Execution / Deployment Layer"]
E1[ContainerRunner<br/>local Docker]
E0[BareMetalRunner<br/>conda]
E2[DeploymentFactory]
K8S[Kubernetes Jobs]
SLURM[SLURM Jobs]
SPUR[Spur job arrays]
LLMD[llm-d]
end
subgraph LAUNCH["Launcher Layer"]
T[Train: torchrun · DeepSpeed<br/>Megatron-LM · TorchTitan · Primus]
I[Infer: vLLM · SGLang · SGLang Disagg]
end
OUT[(perf.csv / JSON)]
C1 --> O1
C2 --> O2
C3 --> O3
O2 --> MAN --> O3
O1 --> O2
O3 --> E1
O3 --> E0
O3 --> E2
E2 --> K8S
E2 --> SLURM
E2 --> SPUR
E2 --> LLMD
LLMD --> OUT
E0 --> OUT
E1 --> LAUNCH
K8S --> LAUNCH
SLURM --> LAUNCH
SPUR --> LAUNCH
LAUNCH --> OUT
OUT --> C4
OUT --> C5
- CLI Layer - User interface with 7 commands (discover, build, run, report, database, cleanup, setup-page)
- Model Discovery - Find and validate models from MAD package
- Orchestration - BuildOrchestrator & RunOrchestrator manage workflows
- Execution Targets - Local Docker, bare-metal conda, Kubernetes Jobs, SLURM Jobs, Spur job arrays, or llm-d
- Distributed Launchers - Training (torchrun, DeepSpeed, Megatron-LM, TorchTitan, Primus) and Inference (vLLM, SGLang)
- Performance Output - CSV/JSON results with metrics
- Post-Processing - Report generation (HTML/Email) and database upload (MongoDB)
- Main Repository: https://github.com/ROCm/madengine
- MAD Package: https://github.com/ROCm/MAD
- Issues: https://github.com/ROCm/madengine/issues
- ROCm Documentation: https://rocm.docs.amd.com/
Run a model locally → Installation → Usage Guide
Run on this machine without Docker → Bare metal
Deploy to Kubernetes → Configuration → Deployment
Deploy to SLURM or Spur → Configuration → Deployment
Build multiple models selectively (CI/CD) → Batch Build
Profile model performance → Profiling
Multi-node distributed training → Launchers → Deployment
Contribute to madengine → Contributing
madengine operates within the MAD (Model Automation and Dashboarding) ecosystem. The MAD package contains:
- Model definitions (
models.json) - Execution scripts (
run.sh) - Docker configurations
- Data provider configurations (
data.json) - Credentials (
credential.json)
madengine - Modern CLI with:
- Rich terminal output
- Distributed deployment support (K8s, SLURM)
- Build/run separation
- Manifest-based execution
- Local Docker - Containers on this machine
- Bare metal - A conda prefix on this machine, with no Docker. See Bare metal
- Kubernetes - Cloud-native container orchestration
- SLURM - HPC cluster job scheduling
- Spur - Crusoe's SLURM-compatible scheduler. Multi-node runs are a job array plus a shared-filesystem rendezvous. See Deployment
- llm-d - Distributed inference stack, benchmarked from a Kubernetes Job
- torchrun - PyTorch DDP/FSDP
- deepspeed - ZeRO optimization
- megatron-lm - Large transformers (K8s + SLURM)
- torchtitan - LLM pre-training
- vllm - LLM inference
- sglang - Structured generation
This documentation follows these principles:
- Task-oriented - Organized by what users want to accomplish
- Progressive disclosure - Start simple, add complexity as needed
- Examples first - Show working examples before explaining details
- Consistent naming - Files follow simple naming pattern (no prefixes)
- Up-to-date - Tracks the current implementation; see CHANGELOG.md for release history
Documentation improvements are welcome! Please:
- Keep examples working and tested
- Use consistent formatting and style
- Update cross-references when moving content
- Mark deprecated content clearly
- Follow the existing structure
See Contributing Guide for details.
madengine is licensed under the MIT License. See LICENSE for details.