diff --git a/inference/a4x/disaggregated-serving/dynamo/README.md b/inference/a4x/disaggregated-serving/dynamo/README.md index c53cdba0..79c62da5 100644 --- a/inference/a4x/disaggregated-serving/dynamo/README.md +++ b/inference/a4x/disaggregated-serving/dynamo/README.md @@ -1,8 +1,8 @@ # Disaggregated Multi-Node Inference with NVIDIA Dynamo on A4X GKE -This document outlines the steps to deploy and serve Large Language Models (LLMs) using [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) disaggregated inference platform on [A4X GKE Node pools](https://cloud.google.com/kubernetes-engine). +This document outlines the steps to deploy and serve Large Language Models (LLMs) using [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) disaggregated inference platform on [A4X (GB200) GKE Node pools](https://cloud.google.com/kubernetes-engine). -Dynamo provides a disaggregated architecture that separates prefill and decode operations for optimized inference performance, supporting both single-node (4 GPUs) and multi-node NVL72 (72 GPUs) configurations. Dynamo also supports various inference framework backends like [vLLM](https://docs.nvidia.com/dynamo/latest/components/backends/vllm/README.html) and [SGLang](https://docs.nvidia.com/dynamo/latest/components/backends/sglang/README.html). In this recipe, we will focus on serving using the SGLang backend. +Dynamo provides a disaggregated architecture that separates prefill and decode operations for optimized inference performance across multi-node NVL72 domains. Dynamo supports various inference framework backends like [vLLM](https://docs.nvidia.com/dynamo/latest/components/backends/vllm/README.html) and [SGLang](https://docs.nvidia.com/dynamo/latest/components/backends/sglang/README.html). In this recipe, we focus on serving `nvidia/DeepSeek-R1-0528-NVFP4-v2` (`ISL=8192`, `OSL=1024`) using the SGLang backend. ## Table of Contents @@ -14,11 +14,11 @@ Dynamo provides a disaggregated architecture that separates prefill and decode o * [2.3. Connect to your GKE Cluster](#connect-cluster) * [2.4. Create Secrets](#create-secrets) * [2.5. Install Dynamo Platform](#install-platform) - * [2.6. Setup GCS Bucket for GKE ](#setup-gcsfuse) - * [2.7. Build Dynamo Image ](#build-dyanmo-image) + * [2.6. Setup GCS Bucket for GKE](#setup-gcsfuse) + * [2.7. Configure Dynamo Image](#configure-dynamo-image) * [3. Deploy with SGLang Backend](#deploy-sglang) - * [3.1. SGLang Deployment without DeepEP(8 GPUs)](#sglang-wo-deepep) - * [3.2. SGLang Deployment with DeepEP(72 GPUs)](#sglang-deepep) + * [3.1. SGLang Deployment without DeepEP (20 GPUs, 1P4D Low Latency)](#sglang-wo-deepep) + * [3.2. SGLang Deployment with DeepEP (72 GPUs, 10P1D High Throughput)](#sglang-deepep) * [4. Inference Request](#inference-request) * [5. Monitoring and Troubleshooting](#monitoring) * [6. Cleanup](#cleanup) @@ -31,14 +31,15 @@ Dynamo provides a disaggregated architecture that separates prefill and decode o This recipe has been tested with the following configuration: * **GKE Cluster**: - * GPU node pools with [a4x-highgpu-4g](https://docs.cloud.google.com/compute/docs/gpus#gb200-gpus) machines: - * For multi-node deployment: 4 machines with 4 GPUs each (16 GPUs total) + * GPU node pools with [a4x-highgpu-4g](https://docs.cloud.google.com/compute/docs/gpus#gb200-gpus) machines (4 GB200 GPUs per node): + * For `1p4d` low-latency deployment (without DeepEP): 5 machines with 4 GPUs each (20 GPUs total: 1 Prefill worker × 4 GPUs + 4 Decode workers × 4 GPUs) + * For `10p1d` high-throughput deployment (with DeepEP): 18 machines with 4 GPUs each (72 GPUs total: 10 Prefill workers × 4 GPUs + 1 Decode worker × 32 GPUs) * [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) enabled * [Cloud Storage FUSE CSI driver for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cloud-storage-fuse-csi-driver) enabled > [!IMPORTANT] > To prepare the required environment, see the [GKE environment setup guide](../../../../docs/configuring-environment-gke-a4x.md). - + ## 2. Environment Setup (One-Time) @@ -64,8 +65,8 @@ export CLUSTER_NAME= export NAMESPACE=dynamo-cloud export NGC_API_KEY= export HF_TOKEN= -export RELEASE_VERSION=0.7.0 -export GCS_BUCKET= +export RELEASE_VERSION=1.4.2 +export GCS_BUCKET= # Set the project for gcloud commands gcloud config set project $PROJECT_ID @@ -94,7 +95,7 @@ gcloud container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_REGION Create the namespace: ```bash -kubectl create namespace ${NAMESPACE} +kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f - kubectl config set-context --current --namespace=$NAMESPACE ``` @@ -104,19 +105,24 @@ kubectl create secret docker-registry nvcr-secret \ --namespace=${NAMESPACE} \ --docker-server=nvcr.io \ --docker-username='$oauthtoken' \ - --docker-password=${NGC_API_KEY} + --docker-password=${NGC_API_KEY} \ + --dry-run=client -o yaml | kubectl apply -f - ``` Create the secret for the Hugging Face token: ```bash kubectl create secret generic hf-token-secret \ --from-literal=HF_TOKEN=${HF_TOKEN} \ - -n ${NAMESPACE} + -n ${NAMESPACE} \ + --dry-run=client -o yaml | kubectl apply -f - ``` ### 2.5. Install Dynamo Platform (One-Time Setup) +> [!NOTE] +> If you are using a shared cluster where `dynamo-platform` is already installed in `dynamo-cloud`, verify it with `helm list -n ${NAMESPACE}` and `kubectl get pods -n ${NAMESPACE}`, and skip to Step 2.6. + Add the NVIDIA Helm repository: ```bash helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ @@ -124,24 +130,17 @@ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ helm repo update ``` -Fetch the Dynamo Helm charts: +Fetch the Dynamo Platform Helm chart: ```bash -helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz ``` -Install the Dynamo CRDs: -```bash -helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz \ - --namespace default \ - --wait \ - --atomic -``` - -Install the Dynamo Platform with Grove & Kai scheduler enabled: +Install the Dynamo Platform with Grove & KAI scheduler enabled: ```bash helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \ - --namespace ${NAMESPACE} --set grove.enabled=true --set kai-scheduler.enabled=true + --namespace ${NAMESPACE} \ + --set global.grove.install=true \ + --set global.kai-scheduler.install=true ``` Verify the installation: @@ -149,19 +148,19 @@ Verify the installation: kubectl get pods -n ${NAMESPACE} ``` -Wait until all pods show a `Running` status before proceeding. +Wait until all platform pods show a `Running` status before proceeding. ### 2.6. Setup GCS Bucket for GKE (One-Time Setup) -It is recommended to utilize [gcsfuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup) to facilitate model access and mitigate [huggingface rate limiting](https://huggingface.co/docs/hub/en/rate-limits#hub-rate-limits) issues. +It is recommended to utilize [gcsfuse](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cloud-storage-fuse-csi-driver-setup) to facilitate model access and mitigate [Hugging Face rate limiting](https://huggingface.co/docs/hub/en/rate-limits#hub-rate-limits) issues. Find the service account (usually annotated to default): ```bash -kubectl get serviceaccounts ${NAMESPACE} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.iam\.gke\.io/gcp-service-account}{"\n"}{end}' +kubectl get serviceaccounts -n ${NAMESPACE} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.iam\.gke\.io/gcp-service-account}{"\n"}{end}' ``` -Config the service account email: +Configure the service account email: ```bash export SERVICE_ACCOUNT_EMAIL=$(kubectl get serviceaccount/default -n ${NAMESPACE} -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}') ``` @@ -170,7 +169,7 @@ Authorize the service account: ```bash gcloud iam service-accounts add-iam-policy-binding ${SERVICE_ACCOUNT_EMAIL} \ --role roles/iam.workloadIdentityUser \ - --member "serviceAccount:project_id.svc.id.goog[${NAMESPACE}/default]" + --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/default]" ``` Grant read access to the bucket: @@ -180,21 +179,22 @@ gcloud storage buckets add-iam-policy-binding ${GCS_BUCKET} \ --role "roles/storage.objectViewer" ``` -Downloading model files into the gcs bucket and set your gcs bucket name in values.yaml file. +Label the `default` service account so the Dynamo operator grants it Kubernetes discovery RBAC permissions: +```bash +kubectl label serviceaccount default -n ${NAMESPACE} nvidia.com/dynamo-component-pod=true --overwrite +``` - -### 2.7. Build Dynamo Image +Download `nvidia/DeepSeek-R1-0528-NVFP4-v2` model files into your GCS bucket (under `nvidia/DeepSeek-R1-0528-NVFP4-v2`) and pass your GCS bucket name when installing the Helm release. -Follow the [Dynamo container guide](https://github.com/ai-dynamo/dynamo/blob/main/container/README.md) to build the image, then push it to your artifact registry. + +### 2.7. Configure Dynamo Image -Build the image like this: -```bash -docker build -f container/Dockerfile.sglang . -t dynamo-wideep --no-cache --build-arg DYNAMO_VERSION=0.7.0 --platform linux/arm64 -``` +Instead of building the Dynamo container image manually, use the official pre-built SGLang runtime image. -Config the docker image: +For more details, refer to the [NVIDIA Release Artifacts documentation](https://docs.nvidia.com/dynamo/dev/resources/release-artifacts#sglang-runtime). +Set your environment to use the pre-built image: ```bash -export ARTIFACT_REGISTRY= +export DYNAMO_IMAGE="nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13" ``` @@ -202,74 +202,99 @@ export ARTIFACT_REGISTRY= [Back to Top](#table-of-contents) -Deploy Dynamo with SGLang backend for high-performance inference. +Deploy Dynamo with the SGLang backend for high-performance inference using Kubernetes service discovery (`DYN_DISCOVERY_BACKEND=kubernetes`, `DYN_EVENT_PLANE=zmq`, `DYN_REQUEST_PLANE=tcp`) and Mooncake MNNVL/UCX KV cache transfer. -### 3.1. SGLang Deployment without DeepEP (8 GPUs) +### 3.1. SGLang Deployment without DeepEP (20 GPUs, 1P4D Low Latency) -Two nodes deployment uses 8 GPUs across 2 A4X machines, targeting low latency. +The 5-node deployment (`1p4d`) uses 20 GPUs across 5 A4X machines (1 Prefill worker × 4 GPUs TP=4 EP=1 + 4 Decode workers × 4 GPUs TP=4 EP=1 without DeepEP), targeting low latency (`ISL=8192`, `OSL=1024`, `concurrency=4`). -#### DeepSeekR1 671B Model +#### DeepSeek-R1 NVFP4 Model (`1p4d`, 20 GPUs) -Deploy DeepSeekR1-671B across 2 nodes for testing and validation. Note the use of `--set-file prefill_serving_config` and `--set-file decode_serving_config` pointing to the correct model config file. +Deploy `nvidia/DeepSeek-R1-0528-NVFP4-v2` across 5 nodes: ```bash cd $RECIPE_ROOT -helm install -f values_wo_deepep.yaml \ ---set workload.image=${ARTIFACT_REGISTRY} \ ---set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ ---set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml \ ---set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml \ -$USER-dynamo-a4x-1p1d \ -$REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment +helm install $USER-dynamo-a4x-1p4d \ + $REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment \ + -n ${NAMESPACE} -f values_wo_deepep.yaml \ + --set workload.image=${DYNAMO_IMAGE} \ + --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ + --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml \ + --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml ``` -### 3.2. SGLang Deployment with DeepEP (72 GPUs) - -Multi-node deployment uses 72 GPUs across 18 A4X machines, providing increased capacity for larger models or higher throughput. +### 3.2. SGLang Deployment with DeepEP (72 GPUs, 10P1D High Throughput) -#### DeepSeekR1 671B Model +The 18-node deployment (`10p1d`) uses a full 72-GPU NVL72 rack across 18 A4X machines (10 Prefill workers × 4 GPUs TP=4 EP=1 + 1 Decode worker × 32 GPUs across 8 nodes TP=32 DP=32 EP=32 with DeepEP `low_latency`) for high throughput (`ISL=8192`, `OSL=1024`, `concurrency=2048`). -Deploy DeepSeekR1-671B across 18 nodes for production workloads. Note the use of `--set-file prefill_serving_config` and `--set-file decode_serving_config` pointing to the correct model config file for a multi node deployment scenario: +#### DeepSeek-R1 NVFP4 Model (`10p1d`, 72 GPUs) ```bash cd $RECIPE_ROOT -helm install -f values_deepep.yaml \ ---set workload.image=${ARTIFACT_REGISTRY} \ ---set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ ---set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml \ ---set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml \ -$USER-dynamo-a4x-multi-node \ -$REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment +helm install $USER-dynamo-a4x-10p1d \ + $REPO_ROOT/src/helm-charts/a4x/inference-templates/dynamo-deployment \ + -n ${NAMESPACE} -f values_deepep.yaml \ + --set workload.image=${DYNAMO_IMAGE} \ + --set volumes.gcsfuse.bucketName=${GCS_BUCKET} \ + --set-file prefill_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml \ + --set-file decode_serving_config=$REPO_ROOT/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml ``` ## 4. Inference Request + [Back to Top](#table-of-contents) -Check if the pods are in `Running` status before sending inference requests. +Check if the pods are in `Running` status before sending inference requests: ```bash kubectl get pods -n ${NAMESPACE} ``` -We can then deploy the benchmark client and send benchark request. -Deploy the benchmark client like this: +Deploy the benchmark client and send benchmark requests: ```bash kubectl apply -f bench_client.yaml -n ${NAMESPACE} ``` -And send the request like this: +Send the benchmark request via `bench-client` (for `1p4d` low latency, or set `--host $USER-dynamo-a4x-10p1d-frontend` and `--max-concurrency 2048` for `10p1d`): ```bash -kubectl exec -it bench-client -- bash -c "cd /workspace/dynamo/examples/backends/sglang/slurm_jobs/scripts/vllm && python3 -u benchmark_serving.py --host $USER-dynamo-a4x-1p1d-frontend --port 8000 --model deepseek-ai/DeepSeek-R1 --tokenizer deepseek-ai/DeepSeek-R1 --backend 'dynamo' --endpoint /v1/completions --disable-tqdm --dataset-name random --num-prompts 2560 --random-input-len 8192 --random-output-len 1024 --random-range-ratio 0.8 --ignore-eos --request-rate inf --percentile-metrics ttft,tpot,itl,e2el --max-concurrency 512" +kubectl exec -it bench-client -n ${NAMESPACE} -- bash -c "cd /workspace/dynamo/examples/backends/sglang/slurm_jobs/scripts/vllm && python3 -u benchmark_serving.py \ + --host $USER-dynamo-a4x-1p4d-frontend \ + --port 8000 \ + --model nvidia/DeepSeek-R1-0528-NVFP4-v2 \ + --tokenizer nvidia/DeepSeek-R1-0528-NVFP4-v2 \ + --backend 'dynamo' \ + --endpoint /v1/completions \ + --disable-tqdm \ + --dataset-name random \ + --num-prompts 40 \ + --random-input-len 8192 \ + --random-output-len 1024 \ + --random-range-ratio 0 \ + --ignore-eos \ + --request-rate inf \ + --percentile-metrics ttft,tpot,itl,e2el \ + --max-concurrency 4" ``` -Or we can send a benchmark request to a frontend pod like this: +Or send a benchmark request directly from a frontend pod: ```bash -kubectl exec -n ${NAMESPACE} $USER-dynamo-multi-node-serving-frontend -- python3 -u -m sglang.bench_serving --backend sglang-oai-chat --base-url http://localhost:8000 --model "deepseek-ai/DeepSeek-R1" --tokenizer /data/model/deepseek-ai/DeepSeek-R1 --dataset-name random --num-prompts 10240 --random-input-len 8192 --random-range-ratio 0.8 --random-output-len 1024 --max-concurrency 2048 +FRONTEND_POD=$(kubectl get pods -n ${NAMESPACE} -l nvidia.com/dynamo-component-type=frontend -o jsonpath='{.items[0].metadata.name}') +kubectl exec -n ${NAMESPACE} ${FRONTEND_POD} -- python3 -u -m sglang.bench_serving \ + --backend sglang-oai-chat \ + --base-url http://localhost:8000 \ + --model "nvidia/DeepSeek-R1-0528-NVFP4-v2" \ + --tokenizer /data/model/nvidia/DeepSeek-R1-0528-NVFP4-v2 \ + --dataset-name random \ + --num-prompts 20480 \ + --random-input-len 8192 \ + --random-range-ratio 0 \ + --random-output-len 1024 \ + --max-concurrency 2048 ``` @@ -277,34 +302,33 @@ kubectl exec -n ${NAMESPACE} $USER-dynamo-multi-node-serving-frontend -- python3 [Back to Top](#table-of-contents) -View logs for different components (replace with your deployment name): +View logs for different components: -You can find the exact pod name by: +Find the pod names: ```bash kubectl get pods -n ${NAMESPACE} ``` Frontend logs: ```bash -kubectl logs -f deployment/$USER-dynamo-multi-node-serving-frontend -n ${NAMESPACE} +kubectl logs -f -l nvidia.com/dynamo-component-type=frontend -n ${NAMESPACE} ``` Decode worker logs: ```bash -kubectl logs -f deployment/$USER-dynamo-multi-node-serving-decode-worker -n ${NAMESPACE} +kubectl logs -f -l nvidia.com/dynamo-sub-component-type=decode -n ${NAMESPACE} ``` Prefill worker logs: ```bash -kubectl logs -f deployment/$USER-dynamo-multi-node-serving-prefill-worker -n ${NAMESPACE} +kubectl logs -f -l nvidia.com/dynamo-sub-component-type=prefill -n ${NAMESPACE} ``` Common issues: -* **Pods stuck in Pending**: Check if nodes have sufficient resources (especially for multi-node deployments) -* **Model download slow**: Large models like DeepSeekR1 671B can take 30 minutes to download -* **Multi-node issues**: Verify network connectivity between nodes and proper subnet configuration -* **Deepep timeout issue**: Recompile DeepEP to patch NUM_CPU_TIMEOUT_SECS and NUM_TIMEOUT_CYCLES in csrc/kernels/configs.cuh during the image build. +* **Pods stuck in Pending**: Check if nodes have sufficient resources (5 nodes for `1p4d`, 18 nodes for `10p1d`) and reside in the same NVL72 node pool. You can pin a specific node pool via `--set nodeSelector."cloud\.google\.com/gke-nodepool"=`. +* **Model download slow**: Ensure your GCS bucket is in the same region as the GKE cluster and gcsfuse file caching is enabled. +* **Multi-node issues**: Verify network connectivity between nodes and proper RDMA/MNNVL ComputeDomain configuration. ## 6. Cleanup @@ -318,23 +342,17 @@ helm list -n ${NAMESPACE} --filter $USER-dynamo- Uninstall specific deployments: ```bash -helm uninstall $USER-dynamo-multi-node-serving -n ${NAMESPACE} +helm uninstall $USER-dynamo-a4x-1p4d -n ${NAMESPACE} +helm uninstall $USER-dynamo-a4x-10p1d -n ${NAMESPACE} +kubectl delete -f bench_client.yaml -n ${NAMESPACE} --ignore-not-found ``` Uninstall Dynamo platform (if no longer needed): ```bash helm uninstall dynamo-platform -n ${NAMESPACE} -helm uninstall dynamo-crds -n default ``` Delete namespace and secrets: ```bash kubectl delete namespace ${NAMESPACE} ``` - -Clean up downloaded charts: -```bash -rm -f dynamo-crds-${RELEASE_VERSION}.tgz -rm -f dynamo-platform-${RELEASE_VERSION}.tgz -``` - diff --git a/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml b/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml index a68f203a..f36bd36b 100644 --- a/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml +++ b/inference/a4x/disaggregated-serving/dynamo/values_deepep.yaml @@ -1,4 +1,4 @@ -# Copyright 2025 Google LLC +# Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,23 +14,36 @@ dynamo: namespace: dynamo-cloud - releaseVersion: "0.7.0" - deploymentName: dynamo-disagg10p8d + releaseVersion: "1.4.2" + deploymentName: "" computeDomain: name: a4x-domain numNodes: 18 resourceClaimTemplateName: a4x-channel - serviceAccountName: + serviceAccountName: default frontend: - image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.7.0 - replicas: 9 + image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13 + replicas: 8 + envs: + - name: DYN_DISCOVERY_BACKEND + value: "kubernetes" + - name: DYN_EVENT_PLANE + value: "zmq" + - name: DYN_REQUEST_PLANE + value: "tcp" + - name: DYN_TCP_REQUEST_TIMEOUT + value: "600" + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid livenessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 150 failureThreshold: 100 readinessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 300 failureThreshold: 100 @@ -38,68 +51,118 @@ dynamo: nodeCount: 8 replicas: 1 envs: - - name: HF_TOKEN - valueFrom: - secretKeyRef: - name: hf-token-secret - key: HF_TOKEN - - name: HF_HUB_ENABLE_HF_TRANSFER - value: "1" - - name: LD_LIBRARY_PATH - value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" - - name: GLOO_SOCKET_IFNAME - value: eth0 - - name: TP_SOCKET_IFNAME - value: eth0 - - name: DYN_SKIP_SGLANG_LOG_FORMATTING - value: "1" - - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK - value: "256" - - name: MC_TE_METRIC - value: "true" - - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE - value: "100000" - - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT - value: "100000" - - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT - value: "100000" - - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT - value: "1000" - - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM - value: "1" - - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL - value: "True" - - name: MC_FORCE_MNNVL - value: "1" - - name: NCCL_MNNVL_ENABLE - value: "1" - - name: NCCL_CUMEM_ENABLE - value: "1" - - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER - value: "0" - - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK - value: "1" - - name: PYTHONUNBUFFERED - value: "1" + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: LD_LIBRARY_PATH + value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" + - name: UCX_TLS + value: "cuda_ipc,cuda_copy,rc" + - name: UCX_CUDA_IPC_ENABLE_MNNVL + value: "y" + - name: UCX_NET_DEVICES + value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" + - name: UCX_IB_GID_INDEX + value: "3" + - name: UCX_CUDA_COPY_DMABUF + value: "y" + - name: UCX_MEMTYPE_CACHE + value: "n" + - name: UCX_IB_PCI_RELAXED_ORDERING + value: "y" + - name: UCX_LOG_LEVEL + value: "info" + - name: NIXL_NUM_TRANSPORT_THREADS + value: "4" + - name: PYTORCH_ALLOC_CONF + value: "expandable_segments:False" + - name: GLOO_SOCKET_IFNAME + value: eth0 + - name: TP_SOCKET_IFNAME + value: eth0 + - name: PYTHONUNBUFFERED + value: "1" + - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT + value: "1800" + - name: DYN_SKIP_SGLANG_LOG_FORMATTING + value: "1" + - name: DYN_DISCOVERY_BACKEND + value: "kubernetes" + - name: DYN_EVENT_PLANE + value: "zmq" + - name: DYN_REQUEST_PLANE + value: "tcp" + - name: SGLANG_ENABLE_JIT_DEEPGEMM + value: "false" + - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK + value: "512" + - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN + value: "1" + - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2 + value: "1" + - name: SGLANG_MOE_NVFP4_DISPATCH + value: "1" + - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE + value: "100000" + - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT + value: "100000" + - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM + value: "1" + - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL + value: "True" + - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER + value: "0" + - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK + value: "1" + - name: MC_TE_METRIC + value: "true" + - name: MC_FORCE_MNNVL + value: "1" + - name: NCCL_MNNVL_ENABLE + value: "1" + - name: NCCL_CUMEM_ENABLE + value: "1" + - name: NCCL_NVLS_ENABLE + value: "1" + - name: NCCL_VMM_ENABLE + value: "1" + - name: NCCL_DEBUG + value: "INFO" livenessProbe: - initialDelaySeconds: 600 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 readinessProbe: - initialDelaySeconds: 60 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 startupProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 1800 prefillWorker: - nodeCount: 2 - replicas: 5 + nodeCount: 1 + replicas: 10 envs: + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid - name: HF_TOKEN valueFrom: secretKeyRef: @@ -110,15 +173,51 @@ dynamo: - name: LD_LIBRARY_PATH value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" - name: UCX_TLS - value: "^tcp" + value: "cuda_ipc,cuda_copy,rc" + - name: UCX_CUDA_IPC_ENABLE_MNNVL + value: "y" + - name: UCX_NET_DEVICES + value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" + - name: UCX_IB_GID_INDEX + value: "3" + - name: UCX_CUDA_COPY_DMABUF + value: "y" + - name: UCX_MEMTYPE_CACHE + value: "n" + - name: UCX_IB_PCI_RELAXED_ORDERING + value: "y" + - name: UCX_LOG_LEVEL + value: "info" + - name: NIXL_LOG_LEVEL + value: "info" + - name: NIXL_NUM_TRANSPORT_THREADS + value: "4" + - name: PYTORCH_ALLOC_CONF + value: "expandable_segments:False" - name: GLOO_SOCKET_IFNAME value: eth0 - name: TP_SOCKET_IFNAME value: eth0 + - name: PYTHONUNBUFFERED + value: "1" + - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT + value: "1800" - name: DYN_SKIP_SGLANG_LOG_FORMATTING value: "1" - - name: MC_TE_METRIC - value: "true" + - name: DYN_DISCOVERY_BACKEND + value: "kubernetes" + - name: DYN_EVENT_PLANE + value: "zmq" + - name: DYN_REQUEST_PLANE + value: "tcp" + - name: SGLANG_ENABLE_JIT_DEEPGEMM + value: "false" + - name: SGLANG_MOE_NVFP4_DISPATCH + value: "1" + - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN + value: "1" + - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2 + value: "1" - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE value: "100000" - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT @@ -127,34 +226,39 @@ dynamo: value: "100000" - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL value: "True" + - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER + value: "0" + - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK + value: "1" + - name: MC_TE_METRIC + value: "true" - name: MC_FORCE_MNNVL value: "1" - name: NCCL_MNNVL_ENABLE value: "1" - name: NCCL_CUMEM_ENABLE value: "1" - - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER - value: "0" - - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK + - name: NCCL_NVLS_ENABLE value: "1" - - name: PYTHONUNBUFFERED + - name: NCCL_VMM_ENABLE value: "1" + - name: NCCL_DEBUG + value: "INFO" livenessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 readinessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 startupProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 1800 - secrets: ngc: @@ -179,8 +283,8 @@ service: worker: 9090 workload: - model: deepseek-ai/DeepSeek-R1 - image: + model: nvidia/DeepSeek-R1-0528-NVFP4-v2 + image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13 framework: sglang configFile: serving-args.yaml configPath: /workload/configs @@ -193,4 +297,6 @@ network: value: "VERSION" quantizations: - - "fp8" + - "fp4" + +nodeSelector: {} diff --git a/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml b/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml index 5308d69f..74aa12d4 100644 --- a/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml +++ b/inference/a4x/disaggregated-serving/dynamo/values_wo_deepep.yaml @@ -1,4 +1,4 @@ -# Copyright 2025 Google LLC +# Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,87 +14,144 @@ dynamo: namespace: dynamo-cloud - releaseVersion: "0.7.0" - deploymentName: dynamo-disagg1p1d + releaseVersion: "1.4.2" + deploymentName: "" computeDomain: name: a4x-domain - numNodes: 2 + numNodes: 5 resourceClaimTemplateName: a4x-channel - serviceAccountName: + serviceAccountName: default frontend: - image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.7.0 - replicas: 9 + image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13 + replicas: 8 + envs: + - name: DYN_DISCOVERY_BACKEND + value: "kubernetes" + - name: DYN_EVENT_PLANE + value: "zmq" + - name: DYN_REQUEST_PLANE + value: "tcp" + - name: DYN_TCP_REQUEST_TIMEOUT + value: "600" + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid livenessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 150 failureThreshold: 100 readinessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 300 failureThreshold: 100 decodeWorker: nodeCount: 1 - replicas: 1 + replicas: 4 envs: - - name: HF_TOKEN - valueFrom: - secretKeyRef: - name: hf-token-secret - key: HF_TOKEN - - name: HF_HUB_ENABLE_HF_TRANSFER - value: "1" - - name: LD_LIBRARY_PATH - value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" - - name: GLOO_SOCKET_IFNAME - value: eth0 - - name: TP_SOCKET_IFNAME - value: eth0 - - name: PYTHONUNBUFFERED - value: "1" - - name: DYN_SKIP_SGLANG_LOG_FORMATTING - value: "1" - - name: SGLANG_ENABLE_JIT_DEEPGEMM - value: "false" - - name: SGLANG_ENABLE_FLASHINFER_GEMM - value: "1" - - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE - value: "100000" - - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT - value: "100000" - - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT - value: "100000" - - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT - value: "1000" - - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM - value: "1" - - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL - value: "True" - - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER - value: "0" - - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK - value: "1" - - name: MC_TE_METRIC - value: "true" - - name: MC_FORCE_MNNVL - value: "1" - - name: NCCL_MNNVL_ENABLE - value: "1" - - name: NCCL_CUMEM_ENABLE - value: "1" + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + - name: HF_HUB_ENABLE_HF_TRANSFER + value: "1" + - name: LD_LIBRARY_PATH + value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" + - name: UCX_TLS + value: "cuda_ipc,cuda_copy,rc" + - name: UCX_CUDA_IPC_ENABLE_MNNVL + value: "y" + - name: UCX_NET_DEVICES + value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" + - name: UCX_IB_GID_INDEX + value: "3" + - name: UCX_CUDA_COPY_DMABUF + value: "y" + - name: UCX_MEMTYPE_CACHE + value: "n" + - name: UCX_IB_PCI_RELAXED_ORDERING + value: "y" + - name: UCX_LOG_LEVEL + value: "info" + - name: NIXL_NUM_TRANSPORT_THREADS + value: "4" + - name: PYTORCH_ALLOC_CONF + value: "expandable_segments:False" + - name: GLOO_SOCKET_IFNAME + value: eth0 + - name: TP_SOCKET_IFNAME + value: eth0 + - name: PYTHONUNBUFFERED + value: "1" + - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT + value: "1800" + - name: DYN_SKIP_SGLANG_LOG_FORMATTING + value: "1" + - name: DYN_DISCOVERY_BACKEND + value: "kubernetes" + - name: DYN_EVENT_PLANE + value: "zmq" + - name: DYN_REQUEST_PLANE + value: "tcp" + - name: SGLANG_ENABLE_JIT_DEEPGEMM + value: "false" + - name: SGLANG_MOE_NVFP4_DISPATCH + value: "1" + - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN + value: "1" + - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2 + value: "1" + - name: SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK + value: "1024" + - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE + value: "100000" + - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT + value: "100000" + - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT + value: "100000" + - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT + value: "1000" + - name: SGLANG_HACK_SEQ_BOOTSTRAP_ROOM + value: "1" + - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL + value: "True" + - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER + value: "0" + - name: SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK + value: "1" + - name: MC_TE_METRIC + value: "true" + - name: MC_FORCE_MNNVL + value: "1" + - name: NCCL_MNNVL_ENABLE + value: "1" + - name: NCCL_CUMEM_ENABLE + value: "1" + - name: NCCL_NVLS_ENABLE + value: "1" + - name: NCCL_VMM_ENABLE + value: "1" + - name: NCCL_DEBUG + value: "INFO" livenessProbe: - initialDelaySeconds: 600 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 readinessProbe: - initialDelaySeconds: 60 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 startupProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 1800 @@ -102,6 +159,10 @@ dynamo: nodeCount: 1 replicas: 1 envs: + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid - name: HF_TOKEN valueFrom: secretKeyRef: @@ -112,18 +173,50 @@ dynamo: - name: LD_LIBRARY_PATH value: "/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/nvidia/lib64" - name: UCX_TLS - value: "^tcp" + value: "cuda_ipc,cuda_copy,rc" + - name: UCX_CUDA_IPC_ENABLE_MNNVL + value: "y" + - name: UCX_NET_DEVICES + value: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1" + - name: UCX_IB_GID_INDEX + value: "3" + - name: UCX_CUDA_COPY_DMABUF + value: "y" + - name: UCX_MEMTYPE_CACHE + value: "n" + - name: UCX_IB_PCI_RELAXED_ORDERING + value: "y" + - name: UCX_LOG_LEVEL + value: "info" + - name: NIXL_LOG_LEVEL + value: "info" + - name: NIXL_NUM_TRANSPORT_THREADS + value: "4" + - name: PYTORCH_ALLOC_CONF + value: "expandable_segments:False" - name: GLOO_SOCKET_IFNAME value: eth0 - name: TP_SOCKET_IFNAME value: eth0 - name: PYTHONUNBUFFERED value: "1" + - name: TORCH_DISTRIBUTED_DEFAULT_TIMEOUT + value: "1800" - name: DYN_SKIP_SGLANG_LOG_FORMATTING value: "1" + - name: DYN_DISCOVERY_BACKEND + value: "kubernetes" + - name: DYN_EVENT_PLANE + value: "zmq" + - name: DYN_REQUEST_PLANE + value: "tcp" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "false" - - name: SGLANG_ENABLE_FLASHINFER_GEMM + - name: SGLANG_MOE_NVFP4_DISPATCH + value: "1" + - name: SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN + value: "1" + - name: SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2 value: "1" - name: SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE value: "100000" @@ -131,6 +224,8 @@ dynamo: value: "100000" - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT value: "100000" + - name: SGLANG_DECODE_BOOTSTRAP_TIMEOUT + value: "1000" - name: SGLANG_MOONCAKE_CUSTOM_MEM_POOL value: "True" - name: SGLANG_USE_MESSAGE_QUEUE_BROADCASTER @@ -145,22 +240,27 @@ dynamo: value: "1" - name: NCCL_CUMEM_ENABLE value: "1" + - name: NCCL_NVLS_ENABLE + value: "1" + - name: NCCL_VMM_ENABLE + value: "1" + - name: NCCL_DEBUG + value: "INFO" livenessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 readinessProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 60 startupProbe: - initialDelaySeconds: 3000 + initialDelaySeconds: 10 periodSeconds: 60 timeoutSeconds: 30 failureThreshold: 1800 - secrets: ngc: @@ -185,8 +285,8 @@ service: worker: 9090 workload: - model: deepseek-ai/DeepSeek-R1 - image: + model: nvidia/DeepSeek-R1-0528-NVFP4-v2 + image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0-dev.1-cuda13 framework: sglang configFile: serving-args.yaml configPath: /workload/configs @@ -199,4 +299,6 @@ network: value: "VERSION" quantizations: - - "fp8" + - "fp4" + +nodeSelector: {} diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml new file mode 100644 index 00000000..2743d219 --- /dev/null +++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-decode.yaml @@ -0,0 +1,56 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +enable-metrics: true +served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2" +disaggregation-transfer-backend: "mooncake" +disaggregation-bootstrap-port: "30001" +disaggregation-mode: "decode" +host: "0.0.0.0" +port: "9090" +discovery-backend: "kubernetes" +event-plane: "zmq" +request-plane: "tcp" +context-length: 9600 +decode-log-interval: 1000 +trust-remote-code: true +kv-cache-dtype: "fp8_e4m3" +attention-backend: "trtllm_mla" +quantization: "modelopt_fp4" +moe-runner-backend: "flashinfer_cutedsl" +disable-radix-cache: true +disable-chunked-prefix-cache: true +stream-interval: 50 +watchdog-timeout: 1000000 +disable-shared-experts-fusion: true +eplb-algorithm: "deepseek" +mem-fraction-static: 0.83 +max-total-tokens: 524288 +chunked-prefill-size: 24576 +max-running-requests: 16384 +moe-a2a-backend: "deepep" +deepep-mode: "low_latency" +ep-dispatch-algorithm: "static" +ep-num-redundant-experts: 32 +cuda-graph-max-bs: 512 +num-reserved-decode-tokens: 112 +moe-dense-tp-size: 1 +enable-dp-lm-head: true +prefill-round-robin-balance: true +enable-dp-attention: true +fp4-gemm-backend: "flashinfer_cutlass" +tp: 32 +dp: 32 +ep-size: 32 +disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml new file mode 100644 index 00000000..861bea76 --- /dev/null +++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-10p1d-prefill.yaml @@ -0,0 +1,48 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +enable-metrics: true +served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2" +disaggregation-transfer-backend: "mooncake" +disaggregation-bootstrap-port: "30001" +disaggregation-mode: "prefill" +host: "0.0.0.0" +port: "9090" +discovery-backend: "kubernetes" +event-plane: "zmq" +request-plane: "tcp" +context-length: 9600 +decode-log-interval: 1000 +trust-remote-code: true +kv-cache-dtype: "fp8_e4m3" +attention-backend: "trtllm_mla" +quantization: "modelopt_fp4" +moe-runner-backend: "flashinfer_trtllm" +disable-radix-cache: true +disable-chunked-prefix-cache: true +stream-interval: 50 +watchdog-timeout: 1000000 +disable-shared-experts-fusion: true +mem-fraction-static: 0.95 +max-total-tokens: 131072 +max-prefill-tokens: 524288 +chunked-prefill-size: 131072 +max-running-requests: 30000 +load-balance-method: "round_robin" +disable-cuda-graph: true +fp4-gemm-backend: "flashinfer_cutlass" +tp: 4 +dp: 1 +ep-size: 1 +disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml similarity index 56% rename from src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml rename to src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml index ff0f3c47..5debfb85 100644 --- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-decode.yaml +++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-decode.yaml @@ -1,4 +1,4 @@ -# Copyright 2025 Google LLC +# Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,28 +13,33 @@ # limitations under the License. enable-metrics: true -served-model-name: deepseek-ai/DeepSeek-R1 +served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2" +disaggregation-transfer-backend: "mooncake" disaggregation-bootstrap-port: "30001" -disaggregation-mode: decode +disaggregation-mode: "decode" host: "0.0.0.0" port: "9090" -disable-radix-cache: true -tensor-parallel-size: 4 -data-parallel-size: 1 -expert-parallel-size: 1 +discovery-backend: "kubernetes" +event-plane: "zmq" +request-plane: "tcp" +context-length: 9600 +decode-log-interval: 1000 +prefill-round-robin-balance: true trust-remote-code: true +disable-radix-cache: true kv-cache-dtype: "fp8_e4m3" attention-backend: "trtllm_mla" -quantization: "fp8" +quantization: "modelopt_fp4" moe-runner-backend: "flashinfer_trtllm" -watchdog-timeout: "1000000" -context-length: "9600" -mem-fraction-static: "0.95" -chunked-prefill-size: "8192" -cuda-graph-max-bs: "512" -max-running-requests: "512" -scheduler-recv-interval: "10" -enable-flashinfer-allreduce-fusion: true +fp4-gemm-backend: "flashinfer_trtllm" +stream-interval: 50 +watchdog-timeout: 1000000 +mem-fraction-static: 0.95 +chunked-prefill-size: 8192 +cuda-graph-max-bs: 256 +scheduler-recv-interval: 10 enable-symm-mem: true -moe-dense-tp-size: "1" -prefill-round-robin-balance: true \ No newline at end of file +tp: 4 +ep-size: 1 +max-total-tokens: 131072 +disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml similarity index 53% rename from src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml rename to src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml index e42cb117..020d6265 100644 --- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-1p1d-prefill.yaml +++ b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp4-1p4d-prefill.yaml @@ -1,4 +1,4 @@ -# Copyright 2025 Google LLC +# Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,28 +13,35 @@ # limitations under the License. enable-metrics: true -served-model-name: deepseek-ai/DeepSeek-R1 +served-model-name: "nvidia/DeepSeek-R1-0528-NVFP4-v2" +disaggregation-transfer-backend: "mooncake" disaggregation-bootstrap-port: "30001" -disaggregation-mode: prefill +disaggregation-mode: "prefill" host: "0.0.0.0" port: "9090" -disable-radix-cache: true -tensor-parallel-size: 4 -data-parallel-size: 1 -expert-parallel-size: 1 +discovery-backend: "kubernetes" +event-plane: "zmq" +request-plane: "tcp" +context-length: 9600 +decode-log-interval: 1000 trust-remote-code: true +disable-radix-cache: true kv-cache-dtype: "fp8_e4m3" attention-backend: "trtllm_mla" -quantization: "fp8" +quantization: "modelopt_fp4" moe-runner-backend: "flashinfer_trtllm" -watchdog-timeout: "1000000" -context-length: "9600" -mem-fraction-static: "0.95" -max-total-tokens: "32768" -chunked-prefill-size: "24576" -cuda-graph-max-bs: "512" -max-running-requests: "512" -load-balance-method: round_robin -scheduler-recv-interval: "10" -enable-flashinfer-allreduce-fusion: true -moe-dense-tp-size: "1" \ No newline at end of file +fp4-gemm-backend: "flashinfer_trtllm" +stream-interval: 50 +watchdog-timeout: 1000000 +mem-fraction-static: 0.95 +max-total-tokens: 32768 +chunked-prefill-size: 24576 +cuda-graph-max-bs: 256 +max-running-requests: 512 +scheduler-recv-interval: 10 +enable-symm-mem: true +load-balance-method: "round_robin" +dp: 1 +tp: 4 +ep-size: 1 +disaggregation-ib-device: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml deleted file mode 100644 index 4369e1ce..00000000 --- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-decode.yaml +++ /dev/null @@ -1,47 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -served-model-name: deepseek-ai/DeepSeek-R1 -disaggregation-mode: decode -disaggregation-bootstrap-port: "30001" -host: "0.0.0.0" -port: "9090" -trust-remote-code: true -skip-tokenizer-init: true -tp-size: "32" -dp-size: "32" -ep-size: "32" -quantization: "fp8" -enable-dp-attention: true -attention-backend: "trtllm_mla" -kv-cache-dtype: "fp8_e4m3" -disable-radix-cache: true -stream-interval: "50" -decode-log-interval: "1000" -max-running-requests: "8192" -context-length: "9300" -watchdog-timeout: "1000000" -disable-shared-experts-fusion: true -eplb-algorithm: deepseek -mem-fraction-static: "0.82" -chunked-prefill-size: "36864" -moe-a2a-backend: "deepep" -deepep-mode: "low_latency" -ep-dispatch-algorithm: static -moe-dense-tp-size: "1" -enable-dp-lm-head: true -prefill-round-robin-balance: true -ep-num-redundant-experts: "32" -cuda-graph-max-bs: "256" -deepep-config: '{"normal_dispatch": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 28,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 256}, "normal_combine": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 15,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 128}}' \ No newline at end of file diff --git a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml b/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml deleted file mode 100644 index 9c86f420..00000000 --- a/src/frameworks/a4x/dynamo-configs/deepseekr1-fp8-10p8d-prefill.yaml +++ /dev/null @@ -1,46 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -served-model-name: deepseek-ai/DeepSeek-R1 -disaggregation-mode: prefill -disaggregation-bootstrap-port: "30001" -host: "0.0.0.0" -port: "9090" -trust-remote-code: true -tp-size: "8" -dp-size: "8" -ep-size: "8" -quantization: "fp8" -enable-dp-attention: true -attention-backend: "trtllm_mla" -kv-cache-dtype: "fp8_e4m3" -disable-radix-cache: true -stream-interval: "50" -max-running-requests: "30000" -context-length: "9300" -watchdog-timeout: "1000000" -disable-shared-experts-fusion: true -eplb-algorithm: deepseek -mem-fraction-static: "0.8" -max-total-tokens: "524288" -chunked-prefill-size: "131072" -load-balance-method: round_robin -disable-cuda-graph: true -moe-a2a-backend: deepep -deepep-mode: normal -ep-dispatch-algorithm: "dynamic" -moe-dense-tp-size: "1" -enable-dp-lm-head: true -ep-num-redundant-experts: "32" -deepep-config: '{"normal_dispatch": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 28,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 256}, "normal_combine": {"num_sms": 128,"num_max_nvl_chunked_send_tokens": 15,"num_max_nvl_chunked_recv_tokens": 256,"num_max_rdma_chunked_send_tokens": 6,"num_max_rdma_chunked_recv_tokens": 128}}' \ No newline at end of file diff --git a/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml b/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml index 25a2209e..fb754d93 100644 --- a/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml +++ b/src/helm-charts/a4x/inference-templates/dynamo-deployment/Chart.yaml @@ -17,4 +17,4 @@ name: a4x-dynamo-deployment description: a4x-dynamo-deployment type: application version: 0.1.0 -appVersion: "0.4.0" \ No newline at end of file +appVersion: "1.4.2" \ No newline at end of file diff --git a/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml b/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml index 8002e43a..9a93bf66 100644 --- a/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml +++ b/src/helm-charts/a4x/inference-templates/dynamo-deployment/templates/dynamo-graph-deployment.yaml @@ -15,7 +15,7 @@ apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeployment metadata: - name: {{ .Values.dynamo.deploymentName }} + name: {{ .Values.dynamo.deploymentName | default .Release.Name }} namespace: {{ .Values.dynamo.namespace }} spec: {{- if .Values.workload.framework }} @@ -26,6 +26,10 @@ spec: dynamoNamespace: {{ .Values.dynamo.namespace }} componentType: frontend replicas: {{ .Values.dynamo.frontend.replicas }} + {{- if .Values.dynamo.frontend.envs }} + envs: + {{- toYaml .Values.dynamo.frontend.envs | nindent 8 }} + {{- end }} resources: requests: cpu: "5" @@ -44,6 +48,11 @@ spec: gke-gcsfuse/cache-path: "/gcs-cache" {{- end }} extraPodSpec: + {{- if .Values.dynamo.serviceAccountName }} + serviceAccountName: {{ .Values.dynamo.serviceAccountName }} + {{- end }} + nodeSelector: + kubernetes.io/arch: arm64 tolerations: - key: "kubernetes.io/arch" operator: "Equal" @@ -155,9 +164,14 @@ spec: operator: In values: - arm64 + {{- if .Values.nodeSelector }} + nodeSelector: + {{- toYaml .Values.nodeSelector | nindent 10 }} + {{- end }} mainContainer: securityContext: privileged: true + runAsUser: 0 image: {{ .Values.workload.image }} workingDir: /sgl-workspace/dynamo/components/backends/sglang startupProbe: @@ -180,7 +194,6 @@ spec: echo "--- VERIFYING NCCL ENV VARS IN SHELL ---" env | grep NCCL_ echo "--- END VERIFICATION ---" - pip install hf_transfer ARGS=() if [ -n "$MODEL_PATH" ]; then @@ -324,9 +337,14 @@ spec: operator: In values: - arm64 + {{- if .Values.nodeSelector }} + nodeSelector: + {{- toYaml .Values.nodeSelector | nindent 10 }} + {{- end }} mainContainer: securityContext: privileged: true + runAsUser: 0 stdin: true tty: true image: {{ .Values.workload.image }} @@ -349,7 +367,6 @@ spec: echo "--- VERIFYING NCCL ENV VARS IN SHELL ---" env | grep NCCL_ echo "--- END VERIFICATION ---" - pip install hf_transfer ARGS=() if [ -n "$MODEL_PATH" ]; then