Skip to content

Fix the existing a3-mega inferencing recipes from old fastrak versions - #169

Merged
Chris113113 merged 2 commits into
mainfrom
pirillo/fix_dsr1
Oct 19, 2026
Merged

Chris113113 merged 2 commits into
mainfrom
pirillo/fix_dsr1

Conversation

@Chris113113

@Chris113113 Chris113113 commented Mar 26, 2026 •

Copy link
Copy Markdown
Collaborator

This pull request fixes the GKE A3-Mega inference recipes and Helm charts for deepseek-ai/DeepSeek-R1 (vLLM and SGLang) and meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 (vLLM).

Root Causes

Multi-node and single-node A3-Mega inference deployments failed during pod startup, weight download, or readiness probing for five reasons:

  1. Outdated GPUDirect-TCPXO plugin configuration: The A3-Mega inference charts defaulted to host-mounted plugins or older container images (v1.0.9 and v1.0.12) with NCCL_NET_GDR_LEVEL="SYS", which triggered ncclUnhandledCudaError and ncclInternalError during multi-node communicator initialization.
  2. SGLang loopback binding: The sglang-inference LeaderWorkerSet template launched sglang.launch_server without --host or --port flags, binding only to 127.0.0.1:30000 and causing Kubelet readiness probes and ClusterIP service traffic to fail with Connection refused.
  3. vLLM multi-host Ray entrypoint mismatch: The vllm-inference/multi-host LeaderWorkerSet template invoked /vllm-workspace/ray_init.sh, whereas src/docker/vllm/vllm.Dockerfile sets WORKDIR /workspace and copied ray_init.sh without executable permissions (exit code 126).
  4. HuggingFace Xet CDN read timeouts: Concurrent shard downloads across 8 GPU worker processes per node exceeded the default 10-second huggingface_hub read timeout.
  5. Llama-4 vLLM version and chart mismatches: The Llama-4 A3-Mega recipe referenced vllm/vllm-openai:v0.7.2 (prior to Llama4ForConditionalGeneration support in v0.8.3), pointed to the removed src/helm-charts/a3mega/vllm-inference root path instead of src/helm-charts/a3mega/vllm-inference/single-host, and set --max-model-len=128000 (exceeding single-node H100 80GB KV cache headroom after loading FP8 weights).

Changes

The updates span the A3-Mega Helm templates, recipe values, and Dockerfiles:

  • Update ncclPluginImage to us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/nccl-plugin-gpudirecttcpx-dev:v1.0.18 and rxdmImage to us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.24 across all A3-Mega inference recipes (useHostPlugin: false, ncclBuildType: "12.4", NCCL_NET_GDR_LEVEL: "PIX").
  • Pass --host 0.0.0.0 and --port {{ $root.Values.sglang.service.ports.http }} to sglang.launch_server in src/helm-charts/a3mega/sglang-inference/templates/lws-deployment.yaml, configure tcpxo-daemon as a native sidecar container (restartPolicy: Always), and remove the unsupported subGroupPolicyType field.
  • Mark src/docker/vllm/ray_init.sh executable (0755), populate both /workspace/ray_init.sh and /vllm-workspace/ray_init.sh in src/docker/vllm/vllm.Dockerfile, and invoke bash /workspace/ray_init.sh in src/helm-charts/a3mega/vllm-inference/multi-host/templates/lws-deployment.yaml.
  • Set HF_HUB_ETAG_TIMEOUT="60" and HF_HUB_DOWNLOAD_TIMEOUT="60" across single-host, multi-host, and sglang-inference Helm templates, and quote container image fields so helm lint succeeds with default placeholder values.
  • Update inference/a3mega/llama-4/vllm-serving-gke to vllm/vllm-openai:v0.8.3, set max-model-len: 32768, remove the unsupported trust-remote-code: true entry from vllm.serverArgs, and point the README to src/helm-charts/a3mega/vllm-inference/single-host.

…harts

- Upgrade A3 Mega FasTrak TCPXO plugin and RxDM daemon to LKG (nccl-v1.0.30-external: v1.0.18 / v1.0.24 / buildType 228) and mount at /usr/local/tcpxo across single-host and multi-host vLLM/SGLang Helm templates.

- Set HF_HOME=/ssd on LocalSSD and bound gke-gcsfuse sidecar memory/cache (32Gi limit, max-size-mb:0) to prevent Exit Code 137 node OOMs during 671B weight downloads.

- Align SGLang LeaderWorkerSet template networking (hostNetwork: true, ClusterFirstWithHostNet, GLOO_SOCKET_IFNAME=eth0) and symmetric leader/worker NCCL env vars with vLLM multi-host chart.

- Remove duplicate Service selector keys in multi-host LWS Service templates and fix Llama-4 Maverick FP8 model ID and README instructions.
- Fix vLLM multi-host LWS ray_init.sh path and executable permissions:
  invoke `bash /workspace/ray_init.sh` in `lws-deployment.yaml`, mark
  `src/docker/vllm/ray_init.sh` executable (`100755`), and ensure both
  `/workspace/ray_init.sh` and `/vllm-workspace/ray_init.sh` exist in
  `src/docker/vllm/vllm.Dockerfile`.
- Fix SGLang multi-host LWS bind address: pass `--host 0.0.0.0` and
  `--port {{ $root.Values.sglang.service.ports.http }}` to
  `python3 -m sglang.launch_server` in `sglang-inference/templates/lws-deployment.yaml`
  so the server binds to all interfaces instead of `127.0.0.1:30000`,
  allowing Kubelet readinessProbe and Kubernetes Service routing to succeed.
- Add `HF_HUB_ETAG_TIMEOUT=60` and `HF_HUB_DOWNLOAD_TIMEOUT=60` across
  single-host vLLM, multi-host vLLM, and SGLang Helm templates to prevent
  transient HuggingFace Xet CDN read timeouts during concurrent 8-rank
  safetensors shard downloads.
@Chris113113
Chris113113 marked this pull request as ready for review October 19, 2026 16:51
@Chris113113 Chris113113 changed the title [WIP] Fix the existing a3-mega recipes from old fastrak versions Fix the existing a3-mega inferencing recipes from old fastrak versions Oct 19, 2026
@Chris113113
Chris113113 merged commit d5612eb into main Oct 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants