diff --git a/inference/a4x/disaggregated-serving/dynamo/README.md b/inference/a4x/disaggregated-serving/dynamo/README.md
index c53cdba0..79c62da5 100644
--- a/inference/a4x/disaggregated-serving/dynamo/README.md
+++ b/inference/a4x/disaggregated-serving/dynamo/README.md
@@ -1,8 +1,8 @@
# Disaggregated Multi-Node Inference with NVIDIA Dynamo on A4X GKE
-This document outlines the steps to deploy and serve Large Language Models (LLMs) using [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) disaggregated inference platform on [A4X GKE Node pools](https://cloud.google.com/kubernetes-engine).
+This document outlines the steps to deploy and serve Large Language Models (LLMs) using [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) disaggregated inference platform on [A4X (GB200) GKE Node pools](https://cloud.google.com/kubernetes-engine).
-Dynamo provides a disaggregated architecture that separates prefill and decode operations for optimized inference performance, supporting both single-node (4 GPUs) and multi-node NVL72 (72 GPUs) configurations. Dynamo also supports various inference framework backends like [vLLM](https://docs.nvidia.com/dynamo/latest/components/backends/vllm/README.html) and [SGLang](https://docs.nvidia.com/dynamo/latest/components/backends/sglang/README.html). In this recipe, we will focus on serving using the SGLang backend.
+Dynamo provides a disaggregated architecture that separates prefill and decode operations for optimized inference performance across multi-node NVL72 domains. Dynamo supports various inference framework backends like [vLLM](https://docs.nvidia.com/dynamo/latest/components/backends/vllm/README.html) and [SGLang](https://docs.nvidia.com/dynamo/latest/components/backends/sglang/README.html). In this recipe, we focus on serving `nvidia/DeepSeek-R1-0528-NVFP4-v2` (`ISL=8192`, `OSL=1024`) using the SGLang backend.
## Table of Contents
@@ -14,11 +14,11 @@ Dynamo provides a disaggregated architecture that separates prefill and decode o
* [2.3. Connect to your GKE Cluster](#connect-cluster)
* [2.4. Create Secrets](#create-secrets)
* [2.5. Install Dynamo Platform](#install-platform)
- * [2.6. Setup GCS Bucket for GKE ](#setup-gcsfuse)
- * [2.7. Build Dynamo Image ](#build-dyanmo-image)
+ * [2.6. Setup GCS Bucket for GKE](#setup-gcsfuse)
+ * [2.7. Configure Dynamo Image](#configure-dynamo-image)
* [3. Deploy with SGLang Backend](#deploy-sglang)
- * [3.1. SGLang Deployment without DeepEP(8 GPUs)](#sglang-wo-deepep)
- * [3.2. SGLang Deployment with DeepEP(72 GPUs)](#sglang-deepep)
+ * [3.1. SGLang Deployment without DeepEP (20 GPUs, 1P4D Low Latency)](#sglang-wo-deepep)
+ * [3.2. SGLang Deployment with DeepEP (72 GPUs, 10P1D High Throughput)](#sglang-deepep)
* [4. Inference Request](#inference-request)
* [5. Monitoring and Troubleshooting](#monitoring)
* [6. Cleanup](#cleanup)
@@ -31,14 +31,15 @@ Dynamo provides a disaggregated architecture that separates prefill and decode o
This recipe has been tested with the following configuration:
* **GKE Cluster**:
- * GPU node pools with [a4x-highgpu-4g](https://docs.cloud.google.com/compute/docs/gpus#gb200-gpus) machines:
- * For multi-node deployment: 4 machines with 4 GPUs each (16 GPUs total)
+ * GPU node pools with [a4x-highgpu-4g](https://docs.cloud.google.com/compute/docs/gpus#gb200-gpus) machines (4 GB200 GPUs per node):
+ * For `1p4d` low-latency deployment (without DeepEP): 5 machines with 4 GPUs each (20 GPUs total: 1 Prefill worker × 4 GPUs + 4 Decode workers × 4 GPUs)
+ * For `10p1d` high-throughput deployment (with DeepEP): 18 machines with 4 GPUs each (72 GPUs total: 10 Prefill workers × 4 GPUs + 1 Decode worker × 32 GPUs)
* [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) enabled
* [Cloud Storage FUSE CSI driver for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver) enabled
> [!IMPORTANT]
> To prepare the required environment, see the [GKE environment setup guide](../../../../docs/configuring-environment-gke-a4x.md).
-
+
## 2. Environment Setup (One-Time)
@@ -64,8 +65,8 @@ export CLUSTER_NAME=
export NAMESPACE=dynamo-cloud
export NGC_API_KEY=
export HF_TOKEN=
-export RELEASE_VERSION=0.7.0
-export GCS_BUCKET=
+export RELEASE_VERSION=1.4.2
+export GCS_BUCKET=
# Set the project for gcloud commands
gcloud config set project $PROJECT_ID
@@ -94,7 +95,7 @@ gcloud container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_REGION
Create the namespace:
```bash
-kubectl create namespace ${NAMESPACE}
+kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -
kubectl config set-context --current --namespace=$NAMESPACE
```
@@ -104,19 +105,24 @@ kubectl create secret docker-registry nvcr-secret \
--namespace=${NAMESPACE} \
--docker-server=nvcr.io \
--docker-username='$oauthtoken' \
- --docker-password=${NGC_API_KEY}
+ --docker-password=${NGC_API_KEY} \
+ --dry-run=client -o yaml | kubectl apply -f -
```
Create the secret for the Hugging Face token:
```bash
kubectl create secret generic hf-token-secret \
--from-literal=HF_TOKEN=${HF_TOKEN} \
- -n ${NAMESPACE}
+ -n ${NAMESPACE} \
+ --dry-run=client -o yaml | kubectl apply -f -
```
### 2.5. Install Dynamo Platform (One-Time Setup)
+> [!NOTE]
+> If you are using a shared cluster where `dynamo-platform` is already installed in `dynamo-cloud`, verify it with `helm list -n ${NAMESPACE}` and `kubectl get pods -n ${NAMESPACE}`, and skip to Step 2.6.
+
Add the NVIDIA Helm repository:
```bash
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
@@ -124,24 +130,17 @@ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
helm repo update
```
-Fetch the Dynamo Helm charts:
+Fetch the Dynamo Platform Helm chart:
```bash
-helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz
helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz
```
-Install the Dynamo CRDs:
-```bash
-helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz \
- --namespace default \
- --wait \
- --atomic
-```
-
-Install the Dynamo Platform with Grove & Kai scheduler enabled:
+Install the Dynamo Platform with Grove & KAI scheduler enabled:
```bash
helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \
- --namespace ${NAMESPACE} --set grove.enabled=true --set kai-scheduler.enabled=true
+ --namespace ${NAMESPACE} \
+ --set global.grove.install=true \
+ --set global.kai-scheduler.install=true
```
Verify the installation:
@@ -149,19 +148,19 @@ Verify the installation:
kubectl get pods -n ${NAMESPACE}
```
-Wait until all pods show a `Running` status before proceeding.
+Wait until all platform pods show a `Running` status before proceeding.
### 2.6. Setup GCS Bucket for GKE (One-Time Setup)
-It is recommended to utilize [gcsfuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup) to facilitate model access and mitigate [huggingface rate limiting](https://huggingface.co/docs/hub/en/rate-limits#hub-rate-limits) issues.
+It is recommended to utilize [gcsfuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup) to facilitate model access and mitigate [Hugging Face rate limiting](https://huggingface.co/docs/hub/en/rate-limits#hub-rate-limits) issues.
Find the service account (usually annotated to default):
```bash
-kubectl get serviceaccounts ${NAMESPACE} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.iam\.gke\.io/gcp-service-account}{"\n"}{end}'
+kubectl get serviceaccounts -n ${NAMESPACE} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.iam\.gke\.io/gcp-service-account}{"\n"}{end}'
```
-Config the service account email:
+Configure the service account email:
```bash
export SERVICE_ACCOUNT_EMAIL=$(kubectl get serviceaccount/default -n ${NAMESPACE} -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}')
```
@@ -170,7 +169,7 @@ Authorize the service account:
```bash
gcloud iam service-accounts add-iam-policy-binding ${SERVICE_ACCOUNT_EMAIL} \
--role roles/iam.workloadIdentityUser \
- --member "serviceAccount:project_id.svc.id.goog[${NAMESPACE}/default]"
+ --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/default]"
```
Grant read access to the bucket:
@@ -180,21 +179,22 @@ gcloud storage buckets add-iam-policy-binding ${GCS_BUCKET} \
--role "roles/storage.objectViewer"
```
-Downloading model files into the gcs bucket and set your gcs bucket name in values.yaml file.
+Label the `default` service account so the Dynamo operator grants it Kubernetes discovery RBAC permissions:
+```bash
+kubectl label serviceaccount default -n ${NAMESPACE} nvidia.com/dynamo-component-pod=true --overwrite
+```
-
-### 2.7. Build Dynamo Image
+Download `nvidia/DeepSeek-R1-0528-NVFP4-v2` model files into your GCS bucket (under `nvidia/DeepSeek-R1-0528-NVFP4-v2`) and pass your GCS bucket name when installing the Helm release.
-Follow the [Dynamo container guide](https://github.com/ai-dynamo/dynamo/blob/main/container/README.md) to build the image, then push it to your artifact registry.
+
+### 2.7. Configure Dynamo Image
-Build the image like this:
-```bash
-docker build -f container/Dockerfile.sglang . -t dynamo-wideep --no-cache --build-arg DYNAMO_VERSION=0.7.0 --platform linux/arm64
-```
+Instead of building the Dynamo container image manually, use the official pre-built SGLang runtime image.
-Config the docker image:
+For more details, refer to the [NVIDIA Release Artifacts documentation](https://docs.nvidia.com/dynamo/dev/resources/release-artifacts#sglang-runtime).
+Set your environment to use the pre-built image:
```bash
-export ARTIFACT_REGISTRY=
+export DYNAMO_IMAGE="nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13"
```
@@ -202,74 +202,99 @@ export ARTIFACT_REGISTRY=
[Back to Top](#table-of-contents)
-Deploy Dynamo with SGLang backend for high-performance inference.
+Deploy Dynamo with the SGLang backend for high-performance inference using Kubernetes service discovery (`DYN_DISCOVERY_BACKEND=kubernetes`, `DYN_EVENT_PLANE=zmq`, `DYN_REQUEST_PLANE=tcp`) and Mooncake MNNVL/UCX KV cache transfer.
-### 3.1. SGLang Deployment without DeepEP (8 GPUs)
+### 3.1. SGLang Deployment without DeepEP (20 GPUs, 1P4D Low Latency)
-Two nodes deployment uses 8 GPUs across 2 A4X machines, targeting low latency.
+The 5-node deployment (`1p4d`) uses 20 GPUs across 5 A4X machines (1 Prefill worker × 4 GPUs TP=4 EP=1 + 4 Decode workers × 4 GPUs TP=4 EP=1 without DeepEP), targeting low latency (`ISL=8192`, `OSL=1024`, `concurrency=4`).
-#### DeepSeekR1 671B Model
+#### DeepSeek-R1 NVFP4 Model (`1p4d`, 20 GPUs)
-Deploy DeepSeekR1-671B across 2 nodes for testing and validation. Note the use of `--set-file prefill_serving_config` and `--set-file decode_serving_config` pointing to the correct model config file.
+Deploy `nvidia/DeepSeek-R1-0528-NVFP4-v2` across 5 nodes:
```bash
cd $RECIPE_ROOT
-helm install -f values_wo_deepep.yaml \
---set workload.image=${ARTIFACT_REGISTRY} \
---set volumes.gcsfuse.bucketName=${GCS_BUCKET} \
---set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml \
---set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml \
-$USER-dynamo-a4x-1p1d \
-$REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment
+helm install $USER-dynamo-a4x-1p4d \
+ $REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment \
+ -n ${NAMESPACE} -f values_wo_deepep.yaml \
+ --set workload.image=${DYNAMO_IMAGE} \
+ --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \
+ --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml \
+ --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml
```
-### 3.2. SGLang Deployment with DeepEP (72 GPUs)
-
-Multi-node deployment uses 72 GPUs across 18 A4X machines, providing increased capacity for larger models or higher throughput.
+### 3.2. SGLang Deployment with DeepEP (72 GPUs, 10P1D High Throughput)
-#### DeepSeekR1 671B Model
+The 18-node deployment (`10p1d`) uses a full 72-GPU NVL72 rack across 18 A4X machines (10 Prefill workers × 4 GPUs TP=4 EP=1 + 1 Decode worker × 32 GPUs across 8 nodes TP=32 DP=32 EP=32 with DeepEP `low_latency`) for high throughput (`ISL=8192`, `OSL=1024`, `concurrency=2048`).
-Deploy DeepSeekR1-671B across 18 nodes for production workloads. Note the use of `--set-file prefill_serving_config` and `--set-file decode_serving_config` pointing to the correct model config file for a multi node deployment scenario:
+#### DeepSeek-R1 NVFP4 Model (`10p1d`, 72 GPUs)
```bash
cd $RECIPE_ROOT
-helm install -f values_deepep.yaml \
---set workload.image=${ARTIFACT_REGISTRY} \
---set volumes.gcsfuse.bucketName=${GCS_BUCKET} \
---set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml \
---set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml \
-$USER-dynamo-a4x-multi-node \
-$REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment
+helm install $USER-dynamo-a4x-10p1d \
+ $REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment \
+ -n ${NAMESPACE} -f values_deepep.yaml \
+ --set workload.image=${DYNAMO_IMAGE} \
+ --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \
+ --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml \
+ --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml
```
## 4. Inference Request
+
[Back to Top](#table-of-contents)
-Check if the pods are in `Running` status before sending inference requests.
+Check if the pods are in `Running` status before sending inference requests:
```bash
kubectl get pods -n ${NAMESPACE}
```
-We can then deploy the benchmark client and send benchark request.
-Deploy the benchmark client like this:
+Deploy the benchmark client and send benchmark requests:
```bash
kubectl apply -f bench_client.yaml -n ${NAMESPACE}
```
-And send the request like this:
+Send the benchmark request via `bench-client` (for `1p4d` low latency, or set `--host $USER-dynamo-a4x-10p1d-frontend` and `--max-concurrency 2048` for `10p1d`):
```bash
-kubectl exec -it bench-client -- bash -c "cd /workspace/dynamo/examples/backends/sglang/slurm_jobs/scripts/vllm && python3 -u benchmark_serving.py --host $USER-dynamo-a4x-1p1d-frontend --port 8000 --model deepseek-ai/DeepSeek-R1 --tokenizer deepseek-ai/DeepSeek-R1 --backend 'dynamo' --endpoint /v1/completions --disable-tqdm --dataset-name random --num-prompts 2560 --random-input-len 8192 --random-output-len 1024 --random-range-ratio 0.8 --ignore-eos --request-rate inf --percentile-metrics ttft,tpot,itl,e2el --max-concurrency 512"
+kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c "cd /workspace/dynamo/examples/backends/sglang/slurm_jobs/scripts/vllm && python3 -u benchmark_serving.py \
+ --host $USER-dynamo-a4x-1p4d-frontend \
+ --port 8000 \
+ --model nvidia/DeepSeek-R1-0528-NVFP4-v2 \
+ --tokenizer nvidia/DeepSeek-R1-0528-NVFP4-v2 \
+ --backend 'dynamo' \
+ --endpoint /v1/completions \
+ --disable-tqdm \
+ --dataset-name random \
+ --num-prompts 40 \
+ --random-input-len 8192 \
+ --random-output-len 1024 \
+ --random-range-ratio 0 \
+ --ignore-eos \
+ --request-rate inf \
+ --percentile-metrics ttft,tpot,itl,e2el \
+ --max-concurrency 4"
```
-Or we can send a benchmark request to a frontend pod like this:
+Or send a benchmark request directly from a frontend pod:
```bash
-kubectl exec -n ${NAMESPACE} $USER-dynamo-multi-node-serving-frontend -- python3 -u -m sglang.bench_serving --backend sglang-oai-chat --base-url http://localhost:8000 --model "deepseek-ai/DeepSeek-R1" --tokenizer /data/model/deepseek-ai/DeepSeek-R1 --dataset-name random --num-prompts 10240 --random-input-len 8192 --random-range-ratio 0.8 --random-output-len 1024 --max-concurrency 2048
+FRONTEND_POD=$(kubectl get pods -n ${NAMESPACE} -l nvidia.com/dynamo-component-type=frontend -o jsonpath='{.items[0].metadata.name}')
+kubectl exec -n ${NAMESPACE} ${FRONTEND_POD} -- python3 -u -m sglang.bench_serving \
+ --backend sglang-oai-chat \
+ --base-url http://localhost:8000 \
+ --model "nvidia/DeepSeek-R1-0528-NVFP4-v2" \
+ --tokenizer /data/model/nvidia/DeepSeek-R1-0528-NVFP4-v2 \
+ --dataset-name random \
+ --num-prompts 20480 \
+ --random-input-len 8192 \
+ --random-range-ratio 0 \
+ --random-output-len 1024 \
+ --max-concurrency 2048
```
@@ -277,34 +302,33 @@ kubectl exec -n ${NAMESPACE} $USER-dynamo-multi-node-serving-frontend -- python3
[Back to Top](#table-of-contents)
-View logs for different components (replace with your deployment name):
+View logs for different components:
-You can find the exact pod name by:
+Find the pod names:
```bash
kubectl get pods -n ${NAMESPACE}
```
Frontend logs:
```bash
-kubectl logs -f deployment/$USER-dynamo-multi-node-serving-frontend -n ${NAMESPACE}
+kubectl logs -f -l nvidia.com/dynamo-component-type=frontend -n ${NAMESPACE}
```
Decode worker logs:
```bash
-kubectl logs -f deployment/$USER-dynamo-multi-node-serving-decode-worker -n ${NAMESPACE}
+kubectl logs -f -l nvidia.com/dynamo-sub-component-type=decode -n ${NAMESPACE}
```
Prefill worker logs:
```bash
-kubectl logs -f deployment/$USER-dynamo-multi-node-serving-prefill-worker -n ${NAMESPACE}
+kubectl logs -f -l nvidia.com/dynamo-sub-component-type=prefill -n ${NAMESPACE}
```
Common issues:
-* **Pods stuck in Pending**: Check if nodes have sufficient resources (especially for multi-node deployments)
-* **Model download slow**: Large models like DeepSeekR1 671B can take 30 minutes to download
-* **Multi-node issues**: Verify network connectivity between nodes and proper subnet configuration
-* **Deepep timeout issue**: Recompile DeepEP to patch NUM_CPU_TIMEOUT_SECS and NUM_TIMEOUT_CYCLES in csrc/kernels/configs.cuh during the image build.
+* **Pods stuck in Pending**: Check if nodes have sufficient resources (5 nodes for `1p4d`, 18 nodes for `10p1d`) and reside in the same NVL72 node pool. You can pin a specific node pool via `--set nodeSelector."cloud\.google\.com/gke-nodepool"=`.
+* **Model download slow**: Ensure your GCS bucket is in the same region as the GKE cluster and gcsfuse file caching is enabled.
+* **Multi-node issues**: Verify network connectivity between nodes and proper RDMA/MNNVL ComputeDomain configuration.
## 6. Cleanup
@@ -318,23 +342,17 @@ helm list -n ${NAMESPACE} --filter $USER-dynamo-
Uninstall specific deployments:
```bash
-helm uninstall $USER-dynamo-multi-node-serving -n ${NAMESPACE}
+helm uninstall $USER-dynamo-a4x-1p4d -n ${NAMESPACE}
+helm uninstall $USER-dynamo-a4x-10p1d -n ${NAMESPACE}
+kubectl delete -f bench_client.yaml -n ${NAMESPACE} --ignore-not-found
```
Uninstall Dynamo platform (if no longer needed):
```bash
helm uninstall dynamo-platform -n ${NAMESPACE}
-helm uninstall dynamo-crds -n default
```
Delete namespace and secrets:
```bash
kubectl delete namespace ${NAMESPACE}
```
-
-Clean up downloaded charts:
-```bash
-rm -f dynamo-crds-${RELEASE_VERSION}.tgz
-rm -f dynamo-platform-${RELEASE_VERSION}.tgz
-```
-
diff --git a/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml b/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml
index a68f203a..f36bd36b 100644
--- a/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml
+++ b/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml
@@ -1,4 +1,4 @@
-# Copyright 2025 Google LLC
+# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -14,23 +14,36 @@
dynamo:
namespace: dynamo-cloud
- releaseVersion: "0.7.0"
- deploymentName: dynamo-disagg10p8d
+ releaseVersion: "1.4.2"
+ deploymentName: ""
computeDomain:
name: a4x-domain
numNodes: 18
resourceClaimTemplateName: a4x-channel
- serviceAccountName:
+ serviceAccountName: default
frontend:
- image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.7.0
- replicas: 9
+ image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13
+ replicas: 8
+ envs:
+ - name: DYN_DISCOVERY_BACKEND
+ value: "kubernetes"
+ - name: DYN_EVENT_PLANE
+ value: "zmq"
+ - name: DYN_REQUEST_PLANE
+ value: "tcp"
+ - name: DYN_TCP_REQUEST_TIMEOUT
+ value: "600"
+ - name: POD_UID
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.uid
livenessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 150
failureThreshold: 100
readinessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 300
failureThreshold: 100
@@ -38,68 +51,118 @@ dynamo:
nodeCount: 8
replicas: 1
envs:
- - name: HF_TOKEN
- valueFrom:
- secretKeyRef:
- name: hf-token-secret
- key: HF_TOKEN
- - name: HF_HUB_ENABLE_HF_TRANSFER
- value: "1"
- - name: LD_LIBRARY_PATH
- value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64"
- - name: GLOO_SOCKET_IFNAME
- value: eth0
- - name: TP_SOCKET_IFNAME
- value: eth0
- - name: DYN_SKIP_SGLANG_LOG_FORMATTING
- value: "1"
- - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK
- value: "256"
- - name: MC_TE_METRIC
- value: "true"
- - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE
- value: "100000"
- - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
- value: "100000"
- - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
- value: "100000"
- - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT
- value: "1000"
- - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM
- value: "1"
- - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL
- value: "True"
- - name: MC_FORCE_MNNVL
- value: "1"
- - name: NCCL_MNNVL_ENABLE
- value: "1"
- - name: NCCL_CUMEM_ENABLE
- value: "1"
- - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER
- value: "0"
- - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK
- value: "1"
- - name: PYTHONUNBUFFERED
- value: "1"
+ - name: POD_UID
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.uid
+ - name: HF_TOKEN
+ valueFrom:
+ secretKeyRef:
+ name: hf-token-secret
+ key: HF_TOKEN
+ - name: HF_HUB_ENABLE_HF_TRANSFER
+ value: "1"
+ - name: LD_LIBRARY_PATH
+ value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64"
+ - name: UCX_TLS
+ value: "cuda_ipc,cuda_copy,rc"
+ - name: UCX_CUDA_IPC_ENABLE_MNNVL
+ value: "y"
+ - name: UCX_NET_DEVICES
+ value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1"
+ - name: UCX_IB_GID_INDEX
+ value: "3"
+ - name: UCX_CUDA_COPY_DMABUF
+ value: "y"
+ - name: UCX_MEMTYPE_CACHE
+ value: "n"
+ - name: UCX_IB_PCI_RELAXED_ORDERING
+ value: "y"
+ - name: UCX_LOG_LEVEL
+ value: "info"
+ - name: NIXL_NUM_TRANSPORT_THREADS
+ value: "4"
+ - name: PYTORCH_ALLOC_CONF
+ value: "expandable_segments:False"
+ - name: GLOO_SOCKET_IFNAME
+ value: eth0
+ - name: TP_SOCKET_IFNAME
+ value: eth0
+ - name: PYTHONUNBUFFERED
+ value: "1"
+ - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT
+ value: "1800"
+ - name: DYN_SKIP_SGLANG_LOG_FORMATTING
+ value: "1"
+ - name: DYN_DISCOVERY_BACKEND
+ value: "kubernetes"
+ - name: DYN_EVENT_PLANE
+ value: "zmq"
+ - name: DYN_REQUEST_PLANE
+ value: "tcp"
+ - name: SGLANG_ENABLE_JIT_DEEPGEMM
+ value: "false"
+ - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK
+ value: "512"
+ - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN
+ value: "1"
+ - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2
+ value: "1"
+ - name: SGLANG_MOE_NVFP4_DISPATCH
+ value: "1"
+ - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE
+ value: "100000"
+ - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
+ value: "100000"
+ - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
+ value: "100000"
+ - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT
+ value: "100000"
+ - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM
+ value: "1"
+ - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL
+ value: "True"
+ - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER
+ value: "0"
+ - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK
+ value: "1"
+ - name: MC_TE_METRIC
+ value: "true"
+ - name: MC_FORCE_MNNVL
+ value: "1"
+ - name: NCCL_MNNVL_ENABLE
+ value: "1"
+ - name: NCCL_CUMEM_ENABLE
+ value: "1"
+ - name: NCCL_NVLS_ENABLE
+ value: "1"
+ - name: NCCL_VMM_ENABLE
+ value: "1"
+ - name: NCCL_DEBUG
+ value: "INFO"
livenessProbe:
- initialDelaySeconds: 600
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
readinessProbe:
- initialDelaySeconds: 60
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
startupProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 1800
prefillWorker:
- nodeCount: 2
- replicas: 5
+ nodeCount: 1
+ replicas: 10
envs:
+ - name: POD_UID
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.uid
- name: HF_TOKEN
valueFrom:
secretKeyRef:
@@ -110,15 +173,51 @@ dynamo:
- name: LD_LIBRARY_PATH
value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64"
- name: UCX_TLS
- value: "^tcp"
+ value: "cuda_ipc,cuda_copy,rc"
+ - name: UCX_CUDA_IPC_ENABLE_MNNVL
+ value: "y"
+ - name: UCX_NET_DEVICES
+ value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1"
+ - name: UCX_IB_GID_INDEX
+ value: "3"
+ - name: UCX_CUDA_COPY_DMABUF
+ value: "y"
+ - name: UCX_MEMTYPE_CACHE
+ value: "n"
+ - name: UCX_IB_PCI_RELAXED_ORDERING
+ value: "y"
+ - name: UCX_LOG_LEVEL
+ value: "info"
+ - name: NIXL_LOG_LEVEL
+ value: "info"
+ - name: NIXL_NUM_TRANSPORT_THREADS
+ value: "4"
+ - name: PYTORCH_ALLOC_CONF
+ value: "expandable_segments:False"
- name: GLOO_SOCKET_IFNAME
value: eth0
- name: TP_SOCKET_IFNAME
value: eth0
+ - name: PYTHONUNBUFFERED
+ value: "1"
+ - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT
+ value: "1800"
- name: DYN_SKIP_SGLANG_LOG_FORMATTING
value: "1"
- - name: MC_TE_METRIC
- value: "true"
+ - name: DYN_DISCOVERY_BACKEND
+ value: "kubernetes"
+ - name: DYN_EVENT_PLANE
+ value: "zmq"
+ - name: DYN_REQUEST_PLANE
+ value: "tcp"
+ - name: SGLANG_ENABLE_JIT_DEEPGEMM
+ value: "false"
+ - name: SGLANG_MOE_NVFP4_DISPATCH
+ value: "1"
+ - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN
+ value: "1"
+ - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2
+ value: "1"
- name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE
value: "100000"
- name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
@@ -127,34 +226,39 @@ dynamo:
value: "100000"
- name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL
value: "True"
+ - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER
+ value: "0"
+ - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK
+ value: "1"
+ - name: MC_TE_METRIC
+ value: "true"
- name: MC_FORCE_MNNVL
value: "1"
- name: NCCL_MNNVL_ENABLE
value: "1"
- name: NCCL_CUMEM_ENABLE
value: "1"
- - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER
- value: "0"
- - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK
+ - name: NCCL_NVLS_ENABLE
value: "1"
- - name: PYTHONUNBUFFERED
+ - name: NCCL_VMM_ENABLE
value: "1"
+ - name: NCCL_DEBUG
+ value: "INFO"
livenessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
readinessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
startupProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 1800
-
secrets:
ngc:
@@ -179,8 +283,8 @@ service:
worker: 9090
workload:
- model: deepseek-ai/DeepSeek-R1
- image:
+ model: nvidia/DeepSeek-R1-0528-NVFP4-v2
+ image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13
framework: sglang
configFile: serving-args.yaml
configPath: /workload/configs
@@ -193,4 +297,6 @@ network:
value: "VERSION"
quantizations:
- - "fp8"
+ - "fp4"
+
+nodeSelector: {}
diff --git a/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml b/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml
index 5308d69f..74aa12d4 100644
--- a/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml
+++ b/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml
@@ -1,4 +1,4 @@
-# Copyright 2025 Google LLC
+# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -14,87 +14,144 @@
dynamo:
namespace: dynamo-cloud
- releaseVersion: "0.7.0"
- deploymentName: dynamo-disagg1p1d
+ releaseVersion: "1.4.2"
+ deploymentName: ""
computeDomain:
name: a4x-domain
- numNodes: 2
+ numNodes: 5
resourceClaimTemplateName: a4x-channel
- serviceAccountName:
+ serviceAccountName: default
frontend:
- image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.7.0
- replicas: 9
+ image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13
+ replicas: 8
+ envs:
+ - name: DYN_DISCOVERY_BACKEND
+ value: "kubernetes"
+ - name: DYN_EVENT_PLANE
+ value: "zmq"
+ - name: DYN_REQUEST_PLANE
+ value: "tcp"
+ - name: DYN_TCP_REQUEST_TIMEOUT
+ value: "600"
+ - name: POD_UID
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.uid
livenessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 150
failureThreshold: 100
readinessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 300
failureThreshold: 100
decodeWorker:
nodeCount: 1
- replicas: 1
+ replicas: 4
envs:
- - name: HF_TOKEN
- valueFrom:
- secretKeyRef:
- name: hf-token-secret
- key: HF_TOKEN
- - name: HF_HUB_ENABLE_HF_TRANSFER
- value: "1"
- - name: LD_LIBRARY_PATH
- value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64"
- - name: GLOO_SOCKET_IFNAME
- value: eth0
- - name: TP_SOCKET_IFNAME
- value: eth0
- - name: PYTHONUNBUFFERED
- value: "1"
- - name: DYN_SKIP_SGLANG_LOG_FORMATTING
- value: "1"
- - name: SGLANG_ENABLE_JIT_DEEPGEMM
- value: "false"
- - name: SGLANG_ENABLE_FLASHINFER_GEMM
- value: "1"
- - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE
- value: "100000"
- - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
- value: "100000"
- - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
- value: "100000"
- - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT
- value: "1000"
- - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM
- value: "1"
- - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL
- value: "True"
- - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER
- value: "0"
- - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK
- value: "1"
- - name: MC_TE_METRIC
- value: "true"
- - name: MC_FORCE_MNNVL
- value: "1"
- - name: NCCL_MNNVL_ENABLE
- value: "1"
- - name: NCCL_CUMEM_ENABLE
- value: "1"
+ - name: POD_UID
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.uid
+ - name: HF_TOKEN
+ valueFrom:
+ secretKeyRef:
+ name: hf-token-secret
+ key: HF_TOKEN
+ - name: HF_HUB_ENABLE_HF_TRANSFER
+ value: "1"
+ - name: LD_LIBRARY_PATH
+ value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64"
+ - name: UCX_TLS
+ value: "cuda_ipc,cuda_copy,rc"
+ - name: UCX_CUDA_IPC_ENABLE_MNNVL
+ value: "y"
+ - name: UCX_NET_DEVICES
+ value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1"
+ - name: UCX_IB_GID_INDEX
+ value: "3"
+ - name: UCX_CUDA_COPY_DMABUF
+ value: "y"
+ - name: UCX_MEMTYPE_CACHE
+ value: "n"
+ - name: UCX_IB_PCI_RELAXED_ORDERING
+ value: "y"
+ - name: UCX_LOG_LEVEL
+ value: "info"
+ - name: NIXL_NUM_TRANSPORT_THREADS
+ value: "4"
+ - name: PYTORCH_ALLOC_CONF
+ value: "expandable_segments:False"
+ - name: GLOO_SOCKET_IFNAME
+ value: eth0
+ - name: TP_SOCKET_IFNAME
+ value: eth0
+ - name: PYTHONUNBUFFERED
+ value: "1"
+ - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT
+ value: "1800"
+ - name: DYN_SKIP_SGLANG_LOG_FORMATTING
+ value: "1"
+ - name: DYN_DISCOVERY_BACKEND
+ value: "kubernetes"
+ - name: DYN_EVENT_PLANE
+ value: "zmq"
+ - name: DYN_REQUEST_PLANE
+ value: "tcp"
+ - name: SGLANG_ENABLE_JIT_DEEPGEMM
+ value: "false"
+ - name: SGLANG_MOE_NVFP4_DISPATCH
+ value: "1"
+ - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN
+ value: "1"
+ - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2
+ value: "1"
+ - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK
+ value: "1024"
+ - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE
+ value: "100000"
+ - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
+ value: "100000"
+ - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
+ value: "100000"
+ - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT
+ value: "1000"
+ - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM
+ value: "1"
+ - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL
+ value: "True"
+ - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER
+ value: "0"
+ - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK
+ value: "1"
+ - name: MC_TE_METRIC
+ value: "true"
+ - name: MC_FORCE_MNNVL
+ value: "1"
+ - name: NCCL_MNNVL_ENABLE
+ value: "1"
+ - name: NCCL_CUMEM_ENABLE
+ value: "1"
+ - name: NCCL_NVLS_ENABLE
+ value: "1"
+ - name: NCCL_VMM_ENABLE
+ value: "1"
+ - name: NCCL_DEBUG
+ value: "INFO"
livenessProbe:
- initialDelaySeconds: 600
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
readinessProbe:
- initialDelaySeconds: 60
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
startupProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 1800
@@ -102,6 +159,10 @@ dynamo:
nodeCount: 1
replicas: 1
envs:
+ - name: POD_UID
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.uid
- name: HF_TOKEN
valueFrom:
secretKeyRef:
@@ -112,18 +173,50 @@ dynamo:
- name: LD_LIBRARY_PATH
value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64"
- name: UCX_TLS
- value: "^tcp"
+ value: "cuda_ipc,cuda_copy,rc"
+ - name: UCX_CUDA_IPC_ENABLE_MNNVL
+ value: "y"
+ - name: UCX_NET_DEVICES
+ value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1"
+ - name: UCX_IB_GID_INDEX
+ value: "3"
+ - name: UCX_CUDA_COPY_DMABUF
+ value: "y"
+ - name: UCX_MEMTYPE_CACHE
+ value: "n"
+ - name: UCX_IB_PCI_RELAXED_ORDERING
+ value: "y"
+ - name: UCX_LOG_LEVEL
+ value: "info"
+ - name: NIXL_LOG_LEVEL
+ value: "info"
+ - name: NIXL_NUM_TRANSPORT_THREADS
+ value: "4"
+ - name: PYTORCH_ALLOC_CONF
+ value: "expandable_segments:False"
- name: GLOO_SOCKET_IFNAME
value: eth0
- name: TP_SOCKET_IFNAME
value: eth0
- name: PYTHONUNBUFFERED
value: "1"
+ - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT
+ value: "1800"
- name: DYN_SKIP_SGLANG_LOG_FORMATTING
value: "1"
+ - name: DYN_DISCOVERY_BACKEND
+ value: "kubernetes"
+ - name: DYN_EVENT_PLANE
+ value: "zmq"
+ - name: DYN_REQUEST_PLANE
+ value: "tcp"
- name: SGLANG_ENABLE_JIT_DEEPGEMM
value: "false"
- - name: SGLANG_ENABLE_FLASHINFER_GEMM
+ - name: SGLANG_MOE_NVFP4_DISPATCH
+ value: "1"
+ - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN
+ value: "1"
+ - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2
value: "1"
- name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE
value: "100000"
@@ -131,6 +224,8 @@ dynamo:
value: "100000"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "100000"
+ - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT
+ value: "1000"
- name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL
value: "True"
- name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER
@@ -145,22 +240,27 @@ dynamo:
value: "1"
- name: NCCL_CUMEM_ENABLE
value: "1"
+ - name: NCCL_NVLS_ENABLE
+ value: "1"
+ - name: NCCL_VMM_ENABLE
+ value: "1"
+ - name: NCCL_DEBUG
+ value: "INFO"
livenessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
readinessProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 60
startupProbe:
- initialDelaySeconds: 3000
+ initialDelaySeconds: 10
periodSeconds: 60
timeoutSeconds: 30
failureThreshold: 1800
-
secrets:
ngc:
@@ -185,8 +285,8 @@ service:
worker: 9090
workload:
- model: deepseek-ai/DeepSeek-R1
- image:
+ model: nvidia/DeepSeek-R1-0528-NVFP4-v2
+ image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13
framework: sglang
configFile: serving-args.yaml
configPath: /workload/configs
@@ -199,4 +299,6 @@ network:
value: "VERSION"
quantizations:
- - "fp8"
+ - "fp4"
+
+nodeSelector: {}
diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml
new file mode 100644
index 00000000..2743d219
--- /dev/null
+++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml
@@ -0,0 +1,56 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+enable-metrics: true
+served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2"
+disaggregation-transfer-backend: "mooncake"
+disaggregation-bootstrap-port: "30001"
+disaggregation-mode: "decode"
+host: "0.0.0.0"
+port: "9090"
+discovery-backend: "kubernetes"
+event-plane: "zmq"
+request-plane: "tcp"
+context-length: 9600
+decode-log-interval: 1000
+trust-remote-code: true
+kv-cache-dtype: "fp8_e4m3"
+attention-backend: "trtllm_mla"
+quantization: "modelopt_fp4"
+moe-runner-backend: "flashinfer_cutedsl"
+disable-radix-cache: true
+disable-chunked-prefix-cache: true
+stream-interval: 50
+watchdog-timeout: 1000000
+disable-shared-experts-fusion: true
+eplb-algorithm: "deepseek"
+mem-fraction-static: 0.83
+max-total-tokens: 524288
+chunked-prefill-size: 24576
+max-running-requests: 16384
+moe-a2a-backend: "deepep"
+deepep-mode: "low_latency"
+ep-dispatch-algorithm: "static"
+ep-num-redundant-experts: 32
+cuda-graph-max-bs: 512
+num-reserved-decode-tokens: 112
+moe-dense-tp-size: 1
+enable-dp-lm-head: true
+prefill-round-robin-balance: true
+enable-dp-attention: true
+fp4-gemm-backend: "flashinfer_cutlass"
+tp: 32
+dp: 32
+ep-size: 32
+disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml
new file mode 100644
index 00000000..861bea76
--- /dev/null
+++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml
@@ -0,0 +1,48 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+enable-metrics: true
+served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2"
+disaggregation-transfer-backend: "mooncake"
+disaggregation-bootstrap-port: "30001"
+disaggregation-mode: "prefill"
+host: "0.0.0.0"
+port: "9090"
+discovery-backend: "kubernetes"
+event-plane: "zmq"
+request-plane: "tcp"
+context-length: 9600
+decode-log-interval: 1000
+trust-remote-code: true
+kv-cache-dtype: "fp8_e4m3"
+attention-backend: "trtllm_mla"
+quantization: "modelopt_fp4"
+moe-runner-backend: "flashinfer_trtllm"
+disable-radix-cache: true
+disable-chunked-prefix-cache: true
+stream-interval: 50
+watchdog-timeout: 1000000
+disable-shared-experts-fusion: true
+mem-fraction-static: 0.95
+max-total-tokens: 131072
+max-prefill-tokens: 524288
+chunked-prefill-size: 131072
+max-running-requests: 30000
+load-balance-method: "round_robin"
+disable-cuda-graph: true
+fp4-gemm-backend: "flashinfer_cutlass"
+tp: 4
+dp: 1
+ep-size: 1
+disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml
similarity index 56%
rename from src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml
rename to src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml
index ff0f3c47..5debfb85 100644
--- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml
+++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml
@@ -1,4 +1,4 @@
-# Copyright 2025 Google LLC
+# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -13,28 +13,33 @@
# limitations under the License.
enable-metrics: true
-served-model-name: deepseek-ai/DeepSeek-R1
+served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2"
+disaggregation-transfer-backend: "mooncake"
disaggregation-bootstrap-port: "30001"
-disaggregation-mode: decode
+disaggregation-mode: "decode"
host: "0.0.0.0"
port: "9090"
-disable-radix-cache: true
-tensor-parallel-size: 4
-data-parallel-size: 1
-expert-parallel-size: 1
+discovery-backend: "kubernetes"
+event-plane: "zmq"
+request-plane: "tcp"
+context-length: 9600
+decode-log-interval: 1000
+prefill-round-robin-balance: true
trust-remote-code: true
+disable-radix-cache: true
kv-cache-dtype: "fp8_e4m3"
attention-backend: "trtllm_mla"
-quantization: "fp8"
+quantization: "modelopt_fp4"
moe-runner-backend: "flashinfer_trtllm"
-watchdog-timeout: "1000000"
-context-length: "9600"
-mem-fraction-static: "0.95"
-chunked-prefill-size: "8192"
-cuda-graph-max-bs: "512"
-max-running-requests: "512"
-scheduler-recv-interval: "10"
-enable-flashinfer-allreduce-fusion: true
+fp4-gemm-backend: "flashinfer_trtllm"
+stream-interval: 50
+watchdog-timeout: 1000000
+mem-fraction-static: 0.95
+chunked-prefill-size: 8192
+cuda-graph-max-bs: 256
+scheduler-recv-interval: 10
enable-symm-mem: true
-moe-dense-tp-size: "1"
-prefill-round-robin-balance: true
\ No newline at end of file
+tp: 4
+ep-size: 1
+max-total-tokens: 131072
+disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml
similarity index 53%
rename from src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml
rename to src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml
index e42cb117..020d6265 100644
--- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml
+++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml
@@ -1,4 +1,4 @@
-# Copyright 2025 Google LLC
+# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -13,28 +13,35 @@
# limitations under the License.
enable-metrics: true
-served-model-name: deepseek-ai/DeepSeek-R1
+served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2"
+disaggregation-transfer-backend: "mooncake"
disaggregation-bootstrap-port: "30001"
-disaggregation-mode: prefill
+disaggregation-mode: "prefill"
host: "0.0.0.0"
port: "9090"
-disable-radix-cache: true
-tensor-parallel-size: 4
-data-parallel-size: 1
-expert-parallel-size: 1
+discovery-backend: "kubernetes"
+event-plane: "zmq"
+request-plane: "tcp"
+context-length: 9600
+decode-log-interval: 1000
trust-remote-code: true
+disable-radix-cache: true
kv-cache-dtype: "fp8_e4m3"
attention-backend: "trtllm_mla"
-quantization: "fp8"
+quantization: "modelopt_fp4"
moe-runner-backend: "flashinfer_trtllm"
-watchdog-timeout: "1000000"
-context-length: "9600"
-mem-fraction-static: "0.95"
-max-total-tokens: "32768"
-chunked-prefill-size: "24576"
-cuda-graph-max-bs: "512"
-max-running-requests: "512"
-load-balance-method: round_robin
-scheduler-recv-interval: "10"
-enable-flashinfer-allreduce-fusion: true
-moe-dense-tp-size: "1"
\ No newline at end of file
+fp4-gemm-backend: "flashinfer_trtllm"
+stream-interval: 50
+watchdog-timeout: 1000000
+mem-fraction-static: 0.95
+max-total-tokens: 32768
+chunked-prefill-size: 24576
+cuda-graph-max-bs: 256
+max-running-requests: 512
+scheduler-recv-interval: 10
+enable-symm-mem: true
+load-balance-method: "round_robin"
+dp: 1
+tp: 4
+ep-size: 1
+disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml
deleted file mode 100644
index 4369e1ce..00000000
--- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml
+++ /dev/null
@@ -1,47 +0,0 @@
-# Copyright 2025 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-served-model-name: deepseek-ai/DeepSeek-R1
-disaggregation-mode: decode
-disaggregation-bootstrap-port: "30001"
-host: "0.0.0.0"
-port: "9090"
-trust-remote-code: true
-skip-tokenizer-init: true
-tp-size: "32"
-dp-size: "32"
-ep-size: "32"
-quantization: "fp8"
-enable-dp-attention: true
-attention-backend: "trtllm_mla"
-kv-cache-dtype: "fp8_e4m3"
-disable-radix-cache: true
-stream-interval: "50"
-decode-log-interval: "1000"
-max-running-requests: "8192"
-context-length: "9300"
-watchdog-timeout: "1000000"
-disable-shared-experts-fusion: true
-eplb-algorithm: deepseek
-mem-fraction-static: "0.82"
-chunked-prefill-size: "36864"
-moe-a2a-backend: "deepep"
-deepep-mode: "low_latency"
-ep-dispatch-algorithm: static
-moe-dense-tp-size: "1"
-enable-dp-lm-head: true
-prefill-round-robin-balance: true
-ep-num-redundant-experts: "32"
-cuda-graph-max-bs: "256"
-deepep-config: '{"normal_dispatch": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 28,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 256}, "normal_combine": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 15,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 128}}'
\ No newline at end of file
diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml
deleted file mode 100644
index 9c86f420..00000000
--- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml
+++ /dev/null
@@ -1,46 +0,0 @@
-# Copyright 2025 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-served-model-name: deepseek-ai/DeepSeek-R1
-disaggregation-mode: prefill
-disaggregation-bootstrap-port: "30001"
-host: "0.0.0.0"
-port: "9090"
-trust-remote-code: true
-tp-size: "8"
-dp-size: "8"
-ep-size: "8"
-quantization: "fp8"
-enable-dp-attention: true
-attention-backend: "trtllm_mla"
-kv-cache-dtype: "fp8_e4m3"
-disable-radix-cache: true
-stream-interval: "50"
-max-running-requests: "30000"
-context-length: "9300"
-watchdog-timeout: "1000000"
-disable-shared-experts-fusion: true
-eplb-algorithm: deepseek
-mem-fraction-static: "0.8"
-max-total-tokens: "524288"
-chunked-prefill-size: "131072"
-load-balance-method: round_robin
-disable-cuda-graph: true
-moe-a2a-backend: deepep
-deepep-mode: normal
-ep-dispatch-algorithm: "dynamic"
-moe-dense-tp-size: "1"
-enable-dp-lm-head: true
-ep-num-redundant-experts: "32"
-deepep-config: '{"normal_dispatch": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 28,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 256}, "normal_combine": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 15,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 128}}'
\ No newline at end of file
diff --git a/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml b/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml
index 25a2209e..fb754d93 100644
--- a/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml
+++ b/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml
@@ -17,4 +17,4 @@ name: a4x-dynamo-deployment
description: a4x-dynamo-deployment
type: application
version: 0.1.0
-appVersion: "0.4.0"
\ No newline at end of file
+appVersion: "1.4.2"
\ No newline at end of file
diff --git a/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml b/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml
index 8002e43a..9a93bf66 100644
--- a/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml
+++ b/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml
@@ -15,7 +15,7 @@
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
- name: {{ .Values.dynamo.deploymentName }}
+ name: {{ .Values.dynamo.deploymentName | default .Release.Name }}
namespace: {{ .Values.dynamo.namespace }}
spec:
{{- if .Values.workload.framework }}
@@ -26,6 +26,10 @@ spec:
dynamoNamespace: {{ .Values.dynamo.namespace }}
componentType: frontend
replicas: {{ .Values.dynamo.frontend.replicas }}
+ {{- if .Values.dynamo.frontend.envs }}
+ envs:
+ {{- toYaml .Values.dynamo.frontend.envs | nindent 8 }}
+ {{- end }}
resources:
requests:
cpu: "5"
@@ -44,6 +48,11 @@ spec:
gke-gcsfuse/cache-path: "/gcs-cache"
{{- end }}
extraPodSpec:
+ {{- if .Values.dynamo.serviceAccountName }}
+ serviceAccountName: {{ .Values.dynamo.serviceAccountName }}
+ {{- end }}
+ nodeSelector:
+ kubernetes.io/arch: arm64
tolerations:
- key: "kubernetes.io/arch"
operator: "Equal"
@@ -155,9 +164,14 @@ spec:
operator: In
values:
- arm64
+ {{- if .Values.nodeSelector }}
+ nodeSelector:
+ {{- toYaml .Values.nodeSelector | nindent 10 }}
+ {{- end }}
mainContainer:
securityContext:
privileged: true
+ runAsUser: 0
image: {{ .Values.workload.image }}
workingDir: /sgl-workspace/dynamo/components/backends/sglang
startupProbe:
@@ -180,7 +194,6 @@ spec:
echo "--- VERIFYING NCCL ENV VARS IN SHELL ---"
env | grep NCCL_
echo "--- END VERIFICATION ---"
- pip install hf_transfer
ARGS=()
if [ -n "$MODEL_PATH" ]; then
@@ -324,9 +337,14 @@ spec:
operator: In
values:
- arm64
+ {{- if .Values.nodeSelector }}
+ nodeSelector:
+ {{- toYaml .Values.nodeSelector | nindent 10 }}
+ {{- end }}
mainContainer:
securityContext:
privileged: true
+ runAsUser: 0
stdin: true
tty: true
image: {{ .Values.workload.image }}
@@ -349,7 +367,6 @@ spec:
echo "--- VERIFYING NCCL ENV VARS IN SHELL ---"
env | grep NCCL_
echo "--- END VERIFICATION ---"
- pip install hf_transfer
ARGS=()
if [ -n "$MODEL_PATH" ]; then