Skip to content

Add NVIDIA Nemotron-3-Ultra 550B NVFP4 Dynamo KV aware routing and round robin recipes for A4X-MAX - #290

Merged
syangx39 merged 1 commit into
AI-Hypercomputer:mainfrom
alisachen-google:n3u-kv-routing-a4xmax-dynamo
Oct 26, 2026
Merged

syangx39 merged 1 commit into
AI-Hypercomputer:mainfrom
alisachen-google:n3u-kv-routing-a4xmax-dynamo

Conversation

@alisachen-google

Copy link
Copy Markdown
Contributor

KV-aware vs. round-robin routing for AgentX agentic workload

  • disaggregated-serving/dynamo/nemotron3-ultra-kv-vs-rr/: 8P8D (16 nodes / 64 GPUs, one NVL72 ComputeDomain, mooncake KV transfer over NVLink).
  • aggregated-serving/dynamo/nemotron3-ultra-kv-vs-rr/: 6 workers (6 nodes / 24 GPUs).
  • arm64 serving image (SGLang CUDA 13 + ai-dynamo 1.4.2 + FlashInfer 0.6.18) under src/docker/dynamo-sglang-n3u.
  • Short pointer sections in both parent READMEs and top-level index entries.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive deployment recipes and configurations for serving the NVIDIA Nemotron-3-Ultra 550B (NVFP4) model on A4X-MAX (GB300) GKE nodes. It covers both aggregated (6 workers) and disaggregated (8P8D) serving setups, comparing KV-aware routing against round-robin routing. The review feedback identifies a recurring typo in the environment variable name SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK across all the new Helm values files. This should be corrected to SGLANG_DISABLE_TP_MEMORY_IMBALANCE_CHECK (with an 'm') to ensure SGLang correctly recognizes and applies the setting.

@alisachen-google
alisachen-google force-pushed the n3u-kv-routing-a4xmax-dynamo branch from 2d0b969 to ca71ede Compare October 25, 2026 21:57
@alisachen-google

Copy link
Copy Markdown
Contributor Author

Hi @Chris113113 @syangx39 , could you help review this PR? Thanks!

@syangx39
syangx39 self-requested a review October 25, 2026 23:11
@syangx39

syangx39 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

The KV/RR comparison is structured well: within each layout, the values files differ only in the deployment name and frontend router settings. The worker topology, environment variables, and serving configuration match. I found one concrete observability issue to fix, plus two reproducibility and validation points.

[P2] Enable SGLang metrics in all three serving configurations

Please add enable-metrics: true to the aggregated configuration and both prefill and decode configurations.

Both READMEs instruct users to collect the worker prefix cache-hit rate through /metrics and --server-metrics to explain the KV/RR latency difference. However, SGLang 0.5.16 defaults enable_metrics to False, and Dynamo 1.4.2 registers the SGLang Prometheus metrics only when that flag is enabled.

DYN_SYSTEM_PORT=9090 exposes the Dynamo metrics endpoint; it does not enable the SGLang cache-hit metrics. The operator's forward-pass metrics setting is also separate. As written, the advertised benchmark cannot collect the engine cache-hit metrics needed for that validation.

After adding the flag, please verify that a worker scrape exposes the expected cache-hit metric or cached/prompt token counters.

Nonblocking: use a shared seed for paired KV/RR runs

Please add an explicit, shared --random-seed to the two benchmark commands. The pinned AIPerf version generates a fresh seed when one is omitted, and uses it to select trajectory start positions.

Consequently, using the same command, dataset, and concurrency does not currently ensure the same sampled starting workload. A shared seed—and paired repeats over several seeds if reporting quantitative gains—would improve reproducibility. This does not by itself invalidate the qualitative findings.

Image validation: document and verify the dependency overrides

The Dockerfile installs ai-dynamo[sglang]==1.4.2, which pins SGLang 0.5.16, then force-installs flashinfer-python==0.6.18 with --no-deps.

This overrides SGLang's declared FlashInfer 0.6.14 pin. There is also a declared dependency conflict between SGLang 0.5.16's nvidia-cutlass-dsl==4.6.0 requirement and FlashInfer 0.6.18's >=4.6.2a0 requirement. The relevant constraints are in Dynamo's pyproject, SGLang's pyproject, and FlashInfer's requirements.

Please document the intended overrides and provide the resolved package manifest, an actual import check, and a small GPU serving smoke test for the resulting image. The current metadata-version check does not exercise imports or kernels. I have not demonstrated a runtime crash from these conflicts, so this is a compatibility-validation gap rather than a confirmed startup failure.

Existing bot comment

The SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK spelling is correct for this dependency version: SGLang 0.5.16 defines that exact name. It should not be changed to IMBALANCE.

Merge recommendation: fix the metrics configuration and confirm that the image built from this exact Dockerfile can serve a request before approval. The shared benchmark seed would also be useful to include in the same update.

Validation scope: static review of the PR, shared chart, and relevant pinned upstream implementations. I did not build the image or run the recipes on a GB300/GKE cluster.

@alisachen-google
alisachen-google force-pushed the n3u-kv-routing-a4xmax-dynamo branch from ca71ede to b16ec7e Compare October 26, 2026 02:15
@alisachen-google

alisachen-google commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @syangx39, addressed in b16ec7e:

  • Metrics: Added enable-metrics: true to all three serving configs, matching the measured runs. Section 4.2 now checks that a worker's /metrics exposes the cache-hit series.
  • Seed: Both benchmark commands use --random-seed 42 (the measured seed). The checklist asks for the same seed per arm and for repeats over several seeds before reporting gains.
  • Image overrides: The Dockerfile and README now document the FlashInfer 0.6.18 and nvidia-cutlass-dsl overrides. The image records pip freeze and pip check output. Section 4.2 adds an in-pod import check and a one-request smoke test.

@alisachen-google
alisachen-google force-pushed the n3u-kv-routing-a4xmax-dynamo branch from b16ec7e to b775628 Compare October 26, 2026 06:25
KV-aware vs. round-robin routing for coding-agent traffic, in its own
nemotron3-ultra-kv-vs-rr/ folder under both A4X-MAX Dynamo recipes:

- disaggregated-serving/dynamo/nemotron3-ultra-kv-vs-rr/: 8P8D (16 nodes /
  64 GPUs, one NVL72 ComputeDomain, mooncake KV transfer over NVLink).
- aggregated-serving/dynamo/nemotron3-ultra-kv-vs-rr/: 6 workers (6 nodes /
  24 GPUs).
- Each folder has a dedicated README (why routing matters, files,
  architecture, deploying both arms, A/B benchmark, router tuning,
  troubleshooting) and paired values files with identical workers:
  *_kv.yaml (tuned KV-aware router) and *_rr.yaml (round-robin baseline).
- Serving configs under src/frameworks/a4xmax/dynamo-configs.
- arm64 serving image (SGLang CUDA 13 + ai-dynamo 1.4.2 + FlashInfer 0.6.18)
  under src/docker/dynamo-sglang-n3u.
- Short pointer sections in both parent READMEs and top-level index entries.
@alisachen-google
alisachen-google force-pushed the n3u-kv-routing-a4xmax-dynamo branch from b775628 to 4a00257 Compare October 26, 2026 06:28
@syangx39
syangx39 merged commit e711ac8 into AI-Hypercomputer:main Oct 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants