Singularity Isaac Sim & Lab

Unofficial instructions for running Isaac Sim and Isaac Lab on a cluster using Singularity/Apptainer.

For Docker, see j3soon/docker-isaac-sim.

The commands use singularity for backward compatibility. You can replace it with apptainer. Run the Bash examples on a Linux x86-64 host with an NVIDIA driver and either container runtime installed.

These versions were tested on Nano4 H200 GPUs with driver 580.65.06:

NGC image	Validated workload
nvcr.io/nvidia/isaac-sim:6.1.0	CUDA kernel and headless GPU PhysX simulation
nvcr.io/nvidia/isaac-lab:3.0.0-rc1	Kit PhysX Cartpole training and checkpoint playback
nvcr.io/nvidia/isaac-lab:3.0.0-rc1-kitless	Newton MJWarp and OVPhysX Cartpole training and checkpoint playback; Newton Warp camera rendering

Isaac Lab 3.0.0-rc1 is an early access release. Its matching source tag is v3.0.0-EA. The Lab images already include the code and Python environment; cloning or installing Isaac Lab is unnecessary for these examples.

Isaac Sim

Retrieve the SIF file from the official registry:

singularity pull isaac-sim_6.1.0.sif docker://nvcr.io/nvidia/isaac-sim:6.1.0

Alternatively, convert an existing local Docker image:

docker pull nvcr.io/nvidia/isaac-sim:6.1.0
singularity build isaac-sim_6.1.0.sif docker-daemon://nvcr.io/nvidia/isaac-sim:6.1.0

On a local PC with a supported RTX GPU and an X11/XWayland desktop, create writable runtime directories and reuse the desktop's Xauthority file to launch the GUI:

mkdir -p sim/home sim/cache/hub sim/cache/kit sim/cache/kit-data
xauth_file=${XAUTHORITY:-$HOME/.Xauthority}
sim=(singularity exec --nv --env "ACCEPT_EULA=Y,XAUTHORITY=/root/.Xauthority"
  --home "$PWD/sim/home:/root"
  --bind "$xauth_file:/root/.Xauthority:ro"
  --bind "$PWD/sim/cache/hub:/var/cache/hub"
  --bind "$PWD/sim/cache/kit:/isaac-sim/kit/cache"
  --bind "$PWD/sim/cache/kit-data:/isaac-sim/kit/data"
  "$PWD/isaac-sim_6.1.0.sif")
"${sim[@]}" /isaac-sim/runapp.sh

# Or run a bundled standalone example on the same RTX host.
"${sim[@]}" /isaac-sim/python.sh -u \
  /isaac-sim/standalone_examples/api/isaacsim.core.experimental.api/simulation_callbacks.py

# Or open a shell to inspect the image.
singularity shell --nv --env ACCEPT_EULA=Y isaac-sim_6.1.0.sif

Isaac Sim 6.1 uses the experimental Core API in its current standalone examples. Older isaacsim.core.api examples moved under /isaac-sim/standalone_examples/deprecated/. The GUI and bundled example above were not part of the H200 validation. Headless mode alone does not disable RTX rendering; scripts for H200 must avoid loading the RTX renderer.

Isaac Lab

Retrieve both container variants:

singularity pull isaac-lab_3.0.0-rc1.sif docker://nvcr.io/nvidia/isaac-lab:3.0.0-rc1
singularity pull isaac-lab_3.0.0-rc1-kitless.sif docker://nvcr.io/nvidia/isaac-lab:3.0.0-rc1-kitless

The Kit image includes Isaac Sim. The kitless image runs without Isaac Sim/Kit and supports the Newton and OVPhysX workflows below.

Create a launcher that uses the image's installed environment and writes caches and logs to a fresh host directory. No writable overlay is needed:

cat > run-lab.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
image=$(realpath "$1")
mode=$2
shift 2
case "$mode" in kit|kitless) ;; *) echo 'Use kit or kitless' >&2; exit 1 ;; esac

runtime=singularity
if command -v apptainer >/dev/null 2>&1; then runtime=apptainer; fi
mkdir -p runs
run_dir=$(mktemp -d "$PWD/runs/isaac-lab.XXXXXX")
mkdir -p "$run_dir"/{home,cache/hub,cache/kit,cache/kit-data,logs}
echo "Run directory: $run_dir"
container=("$runtime" exec --nv --cleanenv
  --home "$run_dir/home:/root"
  --bind "$run_dir/cache/hub:/var/cache/hub"
  --bind "$run_dir/logs:/workspace/isaaclab/logs"
  --env "ACCEPT_EULA=Y,PYTHONDONTWRITEBYTECODE=1,PYTHONUNBUFFERED=1,CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}")
if [[ $mode == kit ]]; then
  container+=(--bind "$run_dir/cache/kit:/isaac-sim/kit/cache")
  container+=(--bind "$run_dir/cache/kit-data:/isaac-sim/kit/data")
fi
exec "${container[@]}" "$image" bash -c \
  'cd /workspace/isaaclab && exec ./isaaclab.sh "$@"' bash "$@"
EOF

Run a bounded Cartpole training example on a local GPU, or submit these commands through Slurm as shown below:

# Kit: Isaac Sim PhysX.
bash run-lab.sh isaac-lab_3.0.0-rc1.sif kit \
  train --rl_library rsl_rl --task Isaac-Cartpole \
  --num_envs 256 --max_iterations 32 --seed 42 --logger tensorboard \
  --experiment_name isaacsim_physx physics=isaacsim_physx

# Kitless: Newton with MuJoCo-Warp.
bash run-lab.sh isaac-lab_3.0.0-rc1-kitless.sif kitless \
  train --rl_library rsl_rl --task Isaac-Cartpole \
  --num_envs 256 --max_iterations 32 --seed 42 --logger tensorboard \
  --experiment_name newton_mjwarp physics=newton_mjwarp

# Kitless: OVPhysX.
bash run-lab.sh isaac-lab_3.0.0-rc1-kitless.sif kitless \
  train --rl_library rsl_rl --task Isaac-Cartpole \
  --num_envs 256 --max_iterations 32 --seed 42 --logger tensorboard \
  --experiment_name ovphysx physics=ovphysx

These training commands run headlessly, so omit the old --headless flag. Task names and entry points have also changed: use Isaac-Cartpole and ./isaaclab.sh train, rather than the old Isaac-Cartpole-v0 and training script path. The bundled isaaclab.sh selects the correct Python environment in each image.

For a bounded Newton camera example, use the kitless image. zero_agent enables a Newton viewer by default, so pass --visualizer none on compute nodes:

bash run-lab.sh isaac-lab_3.0.0-rc1-kitless.sif kitless \
  zero_agent --visualizer none --task Isaac-Cartpole-Camera-Direct --num_envs 4 --max_steps 60 \
  physics=newton_mjwarp renderer=newton_renderer presets=rgb

This selects the Newton Warp renderer without opening a viewport. The H200 camera validation separately checked four 96×96 RGB cameras over 60 steps, with CUDA image buffers and changing, nonblank frames.

Cloud Cluster

Example: NCHC Nano4 Cluster

H200 GPUs have no RT cores, so Isaac Sim's RTX rendering is disabled in these examples. Newton Warp rendering does not require RT cores and passed the kitless camera test on H200. See the Isaac Sim GPU requirements and Isaac Lab renderer selectors.

On your local PC, build the SIF files and create run-lab.sh using the sections above, then upload them:

NCHC_USERNAME='YOUR_NCHC_USERNAME'
ssh "${NCHC_USERNAME}@nano4.nchc.org.tw" 'mkdir -p ~/isaac/images'
rsync -avP isaac-sim_6.1.0.sif isaac-lab_3.0.0-rc1.sif \
  isaac-lab_3.0.0-rc1-kitless.sif "${NCHC_USERNAME}@nano4.nchc.org.tw:isaac/images/"
rsync -avP run-lab.sh "${NCHC_USERNAME}@nano4.nchc.org.tw:isaac/"
ssh "${NCHC_USERNAME}@nano4.nchc.org.tw"

In the SSH session, create the batch script. Run GPU workloads on an allocated compute node:

cd ~/isaac
mkdir -p slurm-logs
cat > isaac-lab_nano4.slurm <<'EOF'
#!/usr/bin/env bash
#SBATCH --job-name=isaac-lab-cartpole
#SBATCH --partition=dev
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=00:30:00
#SBATCH --output=slurm-logs/%j.out
#SBATCH --error=slurm-logs/%j.err

set -euo pipefail
cd "$SLURM_SUBMIT_DIR"
if ! command -v apptainer >/dev/null 2>&1 && ! command -v singularity >/dev/null 2>&1; then
  module load singularity
fi
srun bash run-lab.sh "$@"
EOF

NCHC_ACCOUNT_ID='YOUR_NCHC_ACCOUNT_ID'
sbatch --account="$NCHC_ACCOUNT_ID" isaac-lab_nano4.slurm \
  images/isaac-lab_3.0.0-rc1.sif kit \
  train --rl_library rsl_rl --task Isaac-Cartpole \
  --num_envs 256 --max_iterations 32 --seed 42 --logger tensorboard \
  --experiment_name isaacsim_physx physics=isaacsim_physx

sbatch --account="$NCHC_ACCOUNT_ID" isaac-lab_nano4.slurm \
  images/isaac-lab_3.0.0-rc1-kitless.sif kitless \
  train --rl_library rsl_rl --task Isaac-Cartpole \
  --num_envs 256 --max_iterations 32 --seed 42 --logger tensorboard \
  --experiment_name newton_mjwarp physics=newton_mjwarp

# To test OVPhysX, submit the kitless command with both occurrences of
# newton_mjwarp replaced by ovphysx. The same batch script also accepts
# the zero_agent camera arguments from the Isaac Lab section.

View job status, replacing JOB_ID with the ID printed by sbatch:

squeue --me
tail -f slurm-logs/JOB_ID.out
sacct -j JOB_ID --format=JobID,State,ExitCode
find runs -name 'model_*.pt'

Each run prints its host directory. Training logs and checkpoints are under runs/isaac-lab.*/logs/rsl_rl/. The validated Lab runs each completed 32 training iterations with 256 environments, saved a checkpoint, and passed a separate 120-step playback check. These checks establish that the workflows run; they do not measure policy convergence or performance.

References:

• Nano4 User Guide
• Isaac Lab 3.0 container variants
• Isaac Lab cluster deployment