Skip to content

Latest commit

 

History

History
890 lines (658 loc) · 31.6 KB

cli-reference.md

File metadata and controls

890 lines (658 loc) · 31.6 KB

CLI Command Reference

Complete reference for all madengine CLI commands with detailed options and examples.

Table of Contents

Overview

madengine provides a modern CLI for AI model automation and distributed execution. All commands follow a consistent pattern with rich terminal output and comprehensive error handling.

madengine [OPTIONS] COMMAND [ARGS]...

Global Options

These options are available for the main madengine command:

Option Description
--version Show version and exit
--help Show help message and exit

Commands

discover - Discover Available Models

Discover all models available in the MAD package based on specified tags.

Usage:

madengine discover [OPTIONS]

Options:

Option Short Type Default Description
--tags -t TEXT [] Model tags to discover (can specify multiple)
--verbose -v FLAG False Enable verbose logging

Examples:

# Discover all models
madengine discover

# Discover specific models by tag
madengine discover --tags dummy pyt_huggingface_bert

# Multiple tags with comma separation
madengine discover --tags dummy,multi,vllm

# With verbose output
madengine discover --tags model --verbose

# Directory-specific models (scoped tag: {dir}/{model_or_tag})
madengine discover --tags dummy2/model1

# Dynamic models with parameters
madengine discover --tags dummy3/model3:batch_size=512

Discovery Methods:

  1. Root models - From models.json in MAD package root
  2. Directory-specific - From scripts/{dir}/models.json
  3. Dynamic models - Generated by scripts/{dir}/get_models_json.py

build - Build Docker Images

Build Docker images for models, optionally pushing them to a registry.

Usage:

madengine build [OPTIONS]

Options:

Option Short Type Default Description
--tags -t TEXT [] Model tags to build (can specify multiple)
--target-archs -a TEXT [] Target GPU architectures (e.g., gfx908,gfx90a,gfx942)
--registry -r TEXT None Docker registry to push images to
--use-image TEXT None Skip Docker build and use a pre-built image. Pass an image name, or auto to resolve from model card's DOCKER_IMAGE_NAME. Mutually exclusive with --registry and --build-on-compute
--build-on-compute FLAG False Build Docker images on a SLURM compute node and push to registry. Requires --registry
--batch-manifest TEXT None Input batch.json file for batch build mode
--additional-context -c TEXT "{}" Additional context as JSON string
--additional-context-file -f TEXT None File containing additional context JSON; --additional-context is merged over it key by key
--config TEXT None YAML config file and/or Hydra overrides (repeatable). Mutually exclusive with --additional-context / --additional-context-file. See Configuration — YAML config.
--clean-docker-cache FLAG False Rebuild images without using cache
--manifest-output -m TEXT build_manifest.json Output file for build manifest
--summary-output -s TEXT None Output file for build summary JSON
--live-output -l FLAG False Print output in real-time
--verbose -v FLAG False Enable verbose logging

Examples:

# Basic build
madengine build --tags dummy \
  --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}'

# Build with registry
madengine build --tags model \
  --registry docker.io/myorg \
  --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}'

# Build multiple models
madengine build --tags model1 model2 model3 \
  --registry localhost:5000

# Build for multiple GPU architectures
madengine build --tags model \
  --target-archs gfx908 gfx90a gfx942 \
  --registry gcr.io/myproject

# Clean rebuild without cache
madengine build --tags model --clean-docker-cache

# Batch build mode (selective builds)
madengine build --batch-manifest batch.json \
  --registry docker.io/myorg \
  --additional-context-file config.json

# Custom manifest output
madengine build --tags model \
  --manifest-output my_manifest.json \
  --summary-output build_summary.json

# Real-time output with verbose logging
madengine build --tags model --live-output --verbose

# Use a pre-built image (skip Docker build)
madengine build --tags model --use-image lmsysorg/sglang:v0.5.2rc1-rocm700-mi30x

# Auto-detect image from model card's DOCKER_IMAGE_NAME
madengine build --tags model --use-image auto

# Build on SLURM compute node and push to registry
madengine build --tags model --build-on-compute --registry docker.io/myorg

# Build with YAML config (mutually exclusive with --additional-context)
madengine build --tags model --config +build=ci --registry docker.io/myorg

# Build with user YAML file
madengine build --config my_build.yaml --registry docker.io/myorg

Default Values:

The build command applies the following defaults if not specified:

  • gpu_vendor: AMD
  • guest_os: UBUNTU

Example with defaults:

# Equivalent to providing {"gpu_vendor": "AMD", "guest_os": "UBUNTU"}
madengine build --tags dummy

You will see a message indicating which defaults were applied:

ℹ️  Using default values for build configuration:
   • gpu_vendor: AMD (default)
   • guest_os: UBUNTU (default)

💡 To customize, use --additional-context '{"gpu_vendor": "NVIDIA", "guest_os": "CENTOS"}'

Supported Values:

  • gpu_vendor: "AMD" or "NVIDIA"
  • guest_os: "UBUNTU" or "CENTOS"

Batch Build Mode:

When using --batch-manifest, provide a JSON file with selective build configuration:

[
  {
    "model_name": "model1",
    "build_new": true,
    "registry": "docker.io/myorg",
    "registry_image": "custom-namespace/model1"
  },
  {
    "model_name": "model2",
    "build_new": false
  }
]

See Batch Build Guide for details.


run - Execute Models

Run models locally or deploy to Kubernetes/SLURM clusters.

Usage:

madengine run [OPTIONS]

Options:

Option Short Type Default Description
--tags -t TEXT [] Model tags to run (can specify multiple)
--manifest-file -m TEXT "" Build manifest file path (for pre-built images)
--registry -r TEXT None Docker registry URL
--timeout INT -1 Timeout in seconds. -1 means "not passed" and falls through to the model card's timeout, or 7200s if it has none; 0 disables the timeout; a positive value overrides the model card. See Usage — Custom Timeouts.
--additional-context -c TEXT "{}" Additional context as JSON string
--additional-context-file -f TEXT None File containing additional context JSON; --additional-context is merged over it key by key
--config TEXT None YAML config file and/or Hydra overrides (repeatable). Mutually exclusive with --additional-context / --additional-context-file. See Configuration — YAML config.
--keep-alive FLAG False Keep Docker containers alive after run (local Docker only; ignored with a warning on SLURM/K8s)
--keep-model-dir FLAG False Keep model directory after run (local Docker only; ignored with a warning on SLURM/K8s)
--clean-docker-cache FLAG False Rebuild images without using cache (full workflow)
--skip-model-run FLAG False Skip the model script inside each container. The container still starts and pre_scripts still run; only the model script invocation is skipped (status reported as SKIPPED, exit code 0). Combine with --keep-alive to leave a live container for manual exec. Ignored with a warning on SLURM/K8s targets. See Usage — Skip model run.
--require-pinned-image FLAG False Pull registry images by the sha256 digest recorded in the build manifest (repo@sha256:...) instead of by tag, so a tag that moved between build and run fails loudly instead of silently running a different image. Fails immediately — with no tag fallback — if the manifest has no digest for an image. Equivalent to the require_pinned_image additional-context key. See Configuration — Pinned image digests.
--manifest-output TEXT build_manifest.json Output file for build manifest (full workflow)
--summary-output -s TEXT None Output file for summary JSON
--live-output -l FLAG False Print output in real-time
--output -o TEXT perf.csv Performance output file
--ignore-deprecated FLAG False Force run deprecated models
--data-config TEXT data.json Custom data configuration file
--tools-config TEXT ./scripts/common/tools.json Custom tools JSON configuration
--sys-env-details FLAG True Generate system config env details
--force-mirror-local TEXT None Path to force local data mirroring
--disable-skip-gpu-arch FLAG False Disable skipping models based on GPU architecture
--verbose -v FLAG False Enable verbose logging
--cleanup-perf FLAG False Remove intermediate perf_entry files after run (keeps perf.csv and perf_super files)
--wait-for-gpu-memory INT None Before a local run, poll until every GPU reports at most 512 MB of memory in use. 0 checks once and fails if any GPU is over that threshold; a positive value is a wall-clock timeout in seconds. Ignored with a warning on SLURM/K8s.

Examples:

# Local execution
madengine run --tags dummy \
  --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}'

# Custom host ROCm path (when ROCm is not in /opt/rocm, e.g. TheRock or pip install)
madengine run --tags dummy \
  --additional-context '{"MAD_ROCM_PATH": "/path/to/rocm", "gpu_vendor": "AMD", "guest_os": "UBUNTU"}'

# Custom in-container ROCm path (independent from host)
madengine run --tags dummy \
  --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU", "docker_env_vars": {"MAD_ROCM_PATH": "/path/in/container"}}'

# Run with pre-built images (manifest-based)
madengine run --manifest-file build_manifest.json

# Build in this invocation but skip executing containers (CI: images + manifest only)
madengine run --tags model \
  --additional-context '{"gpu_vendor": "AMD", "guest_os": "UBUNTU"}' \
  --skip-model-run

# Local run: wait up to 30s for every GPU to drop to 512 MB used or less.
# Pass 0 to fail immediately if any GPU is over the threshold.
# The flag is ignored on SLURM and Kubernetes.
madengine run --tags model --wait-for-gpu-memory 30

# Multi-GPU with torchrun
madengine run --tags model \
  --additional-context '{
    "gpu_vendor": "AMD",
    "guest_os": "UBUNTU",
    "docker_gpus": "0,1,2,3",
    "distributed": {
      "launcher": "torchrun",
      "nproc_per_node": 4
    }
  }'

# Kubernetes deployment (minimal config)
madengine run --tags model \
  --additional-context '{"k8s": {"gpu_count": 2}}'

# Kubernetes multi-node with vLLM
madengine run --tags model \
  --additional-context '{
    "k8s": {"gpu_count": 8},
    "distributed": {
      "launcher": "vllm",
      "nnodes": 2,
      "nproc_per_node": 4
    }
  }'

# SLURM deployment
madengine run --tags model \
  --additional-context '{
    "slurm": {
      "partition": "gpu",
      "nodes": 4,
      "gpus_per_node": 8
    },
    "distributed": {
      "launcher": "torchtitan",
      "nnodes": 4,
      "nproc_per_node": 8
    }
  }'

# With profiling tools
madengine run --tags model \
  --additional-context '{
    "gpu_vendor": "AMD",
    "guest_os": "UBUNTU",
    "tools": [
      {"name": "rocprof"},
      {"name": "gpu_info_power_profiler"}
    ]
  }'

# Custom timeout (2 hours)
madengine run --tags model --timeout 7200

# No timeout (run indefinitely)
madengine run --tags model --timeout 0

# Keep container alive for debugging
madengine run --tags model --keep-alive --verbose

# Real-time output
madengine run --tags model --live-output

# Custom performance output file
madengine run --tags model --output my_perf_results.csv

# Clean up intermediate perf files after run
madengine run --tags model --cleanup-perf

# Using JSON configuration file
madengine run --tags model \
  --additional-context-file k8s-config.json

# Using YAML config (mutually exclusive with --additional-context)
madengine run --tags model \
  --config scheduler=slurm \
  --config launcher=torchrun \
  --config distributed.nnodes=4

# YAML config with hardware profile
madengine run --tags model \
  --config +profile=mi300x_8gpu \
  --config +env=nccl_debug

# User YAML file with overrides
madengine run --config my_job.yaml --config distributed.nnodes=8

Execution Modes:

  1. Full Workflow - Build + Run (when no manifest exists)
  2. Execution Only - Run only (when manifest-file provided and exists)
  3. Manifest-based - Use pre-built images from manifest

Deployment Targets:

Checked in this order:

  • Bare metal - "platform": "bare_metal". A conda prefix on this machine. Omitted platform stays Docker. See Bare metal
  • llm-d - An llm_d key is present (checked before k8s, because an llm-d config also carries a k8s block). See llm-d
  • Kubernetes - A k8s or kubernetes key is present
  • Spur - A slurm block with scheduler: spur, or deploy: spur next to a slurm block. See Deployment
  • SLURM - A slurm key is present, or the launcher is slurm_multi
  • Local Docker - None of the above

Performance Output:

Results are saved to CSV file (default: perf.csv) with metrics including:

  • Execution time
  • GPU utilization
  • Memory usage
  • Model-specific performance metrics

report - Generate Reports

Generate HTML reports from CSV performance files.

Subcommands

report to-html - Convert CSV to HTML

Convert a single CSV file to HTML table format.

Usage:

madengine report to-html [OPTIONS]

Options:

Option Short Type Required Description
--csv-file-path TEXT Yes Path to the CSV file to convert
--verbose -v FLAG No Enable verbose logging

Examples:

# Convert CSV to HTML
madengine report to-html --csv-file-path perf_entry.csv

# With custom CSV file
madengine report to-html --csv-file-path results/perf_mi300.csv

# Verbose output
madengine report to-html --csv-file-path perf.csv --verbose

Output: Creates {filename}.html in the same directory as the CSV file.


report to-email - Generate Email Report

Convert all CSV files in a directory to a consolidated email-ready HTML report.

Usage:

madengine report to-email [OPTIONS]

Options:

Option Short Type Default Description
--directory --dir TEXT "." Path to directory containing CSV files
--output -o TEXT run_results.html Output HTML filename
--verbose -v FLAG False Enable verbose logging

Examples:

# Generate email report from current directory
madengine report to-email

# Specify directory
madengine report to-email --directory ./results

# Custom output filename
madengine report to-email --dir ./results --output summary.html

# Verbose output
madengine report to-email --directory ./results --verbose

Output: Creates consolidated HTML report suitable for email distribution.


report tracelens - Analyze GPU Traces

Generate TraceLens performance reports from the trace artifacts a run left behind (torch_profiler_output/, rocprof_output/, slurm_results/, k8s_results/).

Requires TraceLens: pip install 'madengine[tracelens]'. --discover-only works without it.

Usage:

madengine report tracelens [OPTIONS]

Options:

Option Type Default Description
--root TEXT "." Directory searched recursively for traces
--output-dir TEXT tracelens_output Directory for generated reports
--mode TEXT auto auto, pytorch, rocprof, pftrace, or collective
--python TEXT Interpreter that has TraceLens installed
--gpu-arch TEXT GPU arch (e.g. MI300X) enabling roofline bound classification
--world-size INTEGER trace count Rank count for the collective report
--max-traces INTEGER 0 Cap traces analyzed per kind (0 means no cap)
--discover-only FLAG False List discovered traces without running TraceLens
--verbose FLAG False Enable verbose logging

Examples:

# Analyze every trace found under the current directory
madengine report tracelens

# See what would be analyzed first
madengine report tracelens --discover-only

# Enable roofline bound classification
madengine report tracelens --gpu-arch MI300X

# Multi-rank collective analysis of a distributed run
madengine report tracelens --root slurm_results --mode collective --world-size 8

Output: Per-trace reports in --output-dir, plus tracelens_summary.csv and tracelens_summary.json.

See the Profiling Guide for which trace formats map to which report.


report tracelens-compare - Diff TraceLens Reports

Compare two or more TraceLens reports into a single diff workbook. The first report is the baseline; every metric gains _diff and _pct columns relative to it.

Usage:

madengine report tracelens-compare REPORT... [OPTIONS]

Options:

Option Short Type Default Description
--output -o TEXT tracelens_comparison.xlsx Output comparison workbook
--names TEXT Display tag per report (repeat the flag)
--python TEXT Interpreter that has TraceLens installed
--verbose -v FLAG False Enable verbose logging

Examples:

# Compare a baseline against a candidate run
madengine report tracelens-compare baseline.xlsx candidate.xlsx

# Label each report and choose the output path
madengine report tracelens-compare a.xlsx b.xlsx \
  --names before --names after -o diff.xlsx

database - Upload to MongoDB

Upload CSV or JSON performance data to MongoDB (format is auto-detected).

Usage:

madengine database [OPTIONS]

Options:

Option Short Type Default Required Description
--file -f TEXT None Yes Path to file (CSV or JSON, auto-detected)
--database --db TEXT None Yes MongoDB database name
--collection -c TEXT None Yes MongoDB collection name
--unique-key -k TEXT None No Unique field(s) for deduplication (comma-separated, auto-detected if not specified)
--batch-size INT 1000 No Batch size for bulk operations
--no-upsert FLAG False No Insert only (don't update existing documents)
--no-index FLAG False No Skip automatic index creation
--dry-run FLAG False No Validate without uploading
--verbose -v FLAG False No Verbose output

Examples:

# Upload JSON with auto-detection
madengine database -f perf_entry_super.json --db mydb -c perf_super

# Upload CSV with custom unique key
madengine database -f perf.csv --db test -c results -k model,timestamp

# Dry run to validate
madengine database -f data.json --db test -c data --dry-run

# With verbose output
madengine database -f perf.csv --db mydb -c results --verbose

Environment Variables:

MongoDB connection details are read from environment variables:

Variable Description Example
MONGO_HOST MongoDB host address (default: localhost) localhost or mongodb.example.com
MONGO_PORT MongoDB port (default: 27017) 27017
MONGO_USER MongoDB username admin
MONGO_PASSWORD MongoDB password secretpassword
MONGO_AUTH_SOURCE MongoDB authentication database (default: admin) admin
MONGO_TIMEOUT_MS Server selection timeout in milliseconds (default: 5000) 5000

Example Setup:

export MONGO_HOST=mongodb.example.com
export MONGO_PORT=27017
export MONGO_USER=myuser
export MONGO_PASSWORD=mypassword

madengine database \
  --file perf_entry.csv \
  --db performance_db \
  --collection model_runs

cleanup - Remove Orphaned Containers

Remove benchmark containers left behind when a run dies before it can clean up. By default only containers labelled madengine.session whose owning madengine process is gone are removed, so this is safe from an if: always() CI step or from cron while other benchmarks are running.

Usage:

madengine cleanup [OPTIONS]

Options:

Option Short Type Default Description
--all FLAG False Also remove containers whose owning madengine process is still running
--dry-run FLAG False List what would be removed, remove nothing

Exit codes: 0 when nothing needs removing or every targeted container was removed. 1 when a container is still present and was not wedged, or when docker ps times out or returns an error — that is not reported as an empty host. 5 (GPU_WEDGED) when a container is still present after stop, kill, and remove, and the host PIDs are stuck in uninterruptible sleep or the daemon still reports the container. A gpu_wedged.json marker is written so CI can take the runner out of rotation.

Local runs also stop themselves. Each container madengine starts runs a POSIX-shell watchdog as PID 1. The watchdog stops the container when that container's heartbeat file in the bind-mounted workspace goes stale. --keep-alive does not install the watchdog, and a cancel leaves that container running. Images without a shell fall back to cat as PID 1 and print a warning that the container is unwatched. A cancel also stops an in-place SLURM step, deletes a submitted SLURM or Kubernetes job, and uninstalls a managed llm-d stack.

Variable Description Default
MADENGINE_HEARTBEAT_INTERVAL Seconds between heartbeat refreshes 15
MADENGINE_HEARTBEAT_STALE_AFTER Seconds without a refresh before the watchdog stops the container 120

A non-integer or non-positive value falls back to the default, and the interval is kept below the staleness threshold.

Examples:

# Sweep orphans after a cancelled CI job
madengine cleanup

# Show what would be removed
madengine cleanup --dry-run

# Also remove containers whose owner is still running
madengine cleanup --all

setup-page - Generate a Setup Picker

Write a self-contained HTML page from the current directory's models.json. The page lists models and context fields and shows the madengine run command to copy. It has no external assets, so it can be published as GitHub Pages. Generation fails if models.json is not in the current directory.

Usage:

madengine setup-page [OPTIONS]

Options:

Option Short Type Default Description
--output -o TEXT index.html Output HTML file path
--title TEXT madengine Setup Picker Page title / heading
--repo-url TEXT None Model repo URL used for clone instructions
--install-cmd TEXT package default madengine install command shown on the page
--verbose -v FLAG False Enable verbose logging

Examples:

# Write index.html from the MAD repo root
madengine setup-page

# Publish under a different name and title
madengine setup-page -o site/index.html --title "MAD setup" \
  --repo-url https://github.com/ROCm/MAD

Exit Codes

madengine uses standard exit codes so scripts and CI (e.g. Jenkins) can detect success or failure:

Code Constant Description
0 SUCCESS Command completed successfully
1 FAILURE General failure
2 BUILD_FAILURE One or more image builds failed (e.g. Docker build error)
3 RUN_FAILURE One or more model executions failed
4 INVALID_ARGS Invalid command-line arguments or configuration
5 GPU_WEDGED madengine cleanup could not remove a container, and the GPU looks wedged. The host needs a reboot before another benchmark

Failure recording: Pre-run failures (e.g. image pull, setup) and run failures are recorded in the performance table (perf.csv) with status FAILURE, so all attempted models appear in the CSV. The file is created automatically if missing.

Example usage in scripts / CI:

#!/bin/bash

madengine build --tags model
if [ $? -eq 0 ]; then
  echo "Build successful"
  madengine run --manifest-file build_manifest.json
else
  echo "Build failed with exit code $?"
  exit $?
fi

Configuration File Format

For complex configurations, use JSON files with --additional-context-file:

Example: config.json

{
  "gpu_vendor": "AMD",
  "guest_os": "UBUNTU",
  "docker_gpus": "0,1,2,3",
  "docker_env_vars": {
    "PYTORCH_TUNABLEOP_ENABLED": "1",
    "HSA_ENABLE_SDMA": "0",
    "NCCL_DEBUG": "INFO"
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 1,
    "nproc_per_node": 4
  }
}

Example: k8s-config.json

{
  "gpu_vendor": "AMD",
  "k8s": {
    "namespace": "ml-team",
    "gpu_count": 8,
    "cpu_request": "32",
    "memory_request": "256Gi",
    "node_selector": {
      "gpu-type": "mi300x"
    }
  },
  "distributed": {
    "launcher": "vllm",
    "nnodes": 2,
    "nproc_per_node": 4
  }
}

Example: slurm-config.json

{
  "gpu_vendor": "AMD",
  "slurm": {
    "partition": "gpu",
    "nodes": 4,
    "gpus_per_node": 8,
    "time": "24:00:00",
    "account": "ml_research",
    "qos": "high"
  },
  "distributed": {
    "launcher": "torchtitan",
    "nnodes": 4,
    "nproc_per_node": 8
  }
}

To run on specific nodes, add "nodelist": "node01,node02" to the slurm section. When set, the job runs only on those nodes and node health preflight is skipped. See examples/slurm-configs/basic/03-multi-node-basic-nodelist.json.

YAML Configuration (--config)

As an alternative to JSON, use --config. After translation it is the same additional-context dict (root slurm / k8s / distributed keys, not a Hydra defaults: list):

# Config group overrides
madengine run --tags model --config scheduler=slurm --config launcher=torchrun

# User YAML file
madengine run --config my_job.yaml

# Append-only groups (profiles, tools, env presets)
madengine run --tags model --config +profile=mi300x_8gpu --config +env=nccl_debug

--config is mutually exclusive with --additional-context / --additional-context-file. See Configuration Guide — YAML Configuration for config groups, user YAML format, and full examples.

Run phase: log error pattern scan (optional)

These keys apply to local Docker runs when madengine post-processes the run log. Use them when substring matches cause false FAILURE status (for example benign RuntimeError: lines). Full details: Configuration — Run phase: log error pattern scan.

Key Description
log_error_pattern_scan Default true. Set false to skip grep-based log failure detection.
log_error_benign_patterns Array of extra strings to exclude from matching (merged with built-in benign list).
log_error_patterns Non-empty array replaces the default substring list (advanced).

Environment Variables

madengine recognizes these environment variables:

Variable Description Default
MODEL_DIR Path to MAD package directory Auto-detected
ROCM_PATH Host ROCm installation root (fallback when MAD_ROCM_PATH is not set in additional context and auto-detect is disabled or finds nothing). In-container ROCM_PATH for Docker is not taken from this variable; set docker_env_vars.MAD_ROCM_PATH in additional context instead. /opt/rocm
MAD_AUTO_ROCM_PATH Set to 0 to disable host auto-detect (ROCM_PATH then /opt/rocm on the host). (default: scan on)
MAD_VERBOSE_CONFIG Enable verbose configuration logging false
MAD_DOCKERHUB_USER Docker Hub username None
MAD_DOCKERHUB_PASSWORD Docker Hub password/token None
MAD_DOCKERHUB_REPO Docker Hub repository None
DOCKER_CONFIG Directory holding the Docker config.json whose existing login madengine reuses ~/.docker
MAD_SKIP_DOCKER_LOGIN Set to 1 to never run docker login; always defer to the machine's existing credentials Unset
MONGO_HOST MongoDB host for database command localhost
MONGO_PORT MongoDB port for database command 27017
MONGO_USER MongoDB username None
MONGO_PASSWORD MongoDB password None
MADENGINE_HEARTBEAT_INTERVAL Seconds between in-container heartbeat refreshes 15
MADENGINE_HEARTBEAT_STALE_AFTER Seconds without a refresh before the watchdog stops the container 120

MAD_CONTAINER_IMAGE is not an environment variable. It is an --additional-context key that selects a pre-built image and skips the build phase — see Configuration.


Best Practices

  1. Use configuration files for complex setups instead of long command lines
  2. Separate build and run phases for distributed deployments
  3. Test locally first before deploying to clusters
  4. Use registries for distributed execution across multiple nodes
  5. Enable verbose logging (--verbose) when debugging issues
  6. Use real-time output (--live-output) for long-running operations
  7. Version your configuration files alongside your model code
  8. Use batch build mode for CI/CD pipelines to optimize build times

Related Documentation


Run madengine --version for the installed version (derived from git tags via versioningit). Release history is in CHANGELOG.md.