Repository navigation
Add NVIDIA Nemotron-3-Ultra 550B NVFP4 Dynamo KV aware routing and round robin recipes for A4X-MAX - #290
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive deployment recipes and configurations for serving the NVIDIA Nemotron-3-Ultra 550B (NVFP4) model on A4X-MAX (GB300) GKE nodes. It covers both aggregated (6 workers) and disaggregated (8P8D) serving setups, comparing KV-aware routing against round-robin routing. The review feedback identifies a recurring typo in the environment variable name SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK across all the new Helm values files. This should be corrected to SGLANG_DISABLE_TP_MEMORY_IMBALANCE_CHECK (with an 'm') to ensure SGLang correctly recognizes and applies the setting.
2d0b969 to
ca71ede
Compare
|
Hi @Chris113113 @syangx39 , could you help review this PR? Thanks! |
|
The KV/RR comparison is structured well: within each layout, the values files differ only in the deployment name and frontend router settings. The worker topology, environment variables, and serving configuration match. I found one concrete observability issue to fix, plus two reproducibility and validation points. [P2] Enable SGLang metrics in all three serving configurationsPlease add Both READMEs instruct users to collect the worker prefix cache-hit rate through
After adding the flag, please verify that a worker scrape exposes the expected cache-hit metric or cached/prompt token counters. Nonblocking: use a shared seed for paired KV/RR runsPlease add an explicit, shared Consequently, using the same command, dataset, and concurrency does not currently ensure the same sampled starting workload. A shared seed—and paired repeats over several seeds if reporting quantitative gains—would improve reproducibility. This does not by itself invalidate the qualitative findings. Image validation: document and verify the dependency overridesThe Dockerfile installs This overrides SGLang's declared FlashInfer 0.6.14 pin. There is also a declared dependency conflict between SGLang 0.5.16's Please document the intended overrides and provide the resolved package manifest, an actual import check, and a small GPU serving smoke test for the resulting image. The current metadata-version check does not exercise imports or kernels. I have not demonstrated a runtime crash from these conflicts, so this is a compatibility-validation gap rather than a confirmed startup failure. Existing bot commentThe Merge recommendation: fix the metrics configuration and confirm that the image built from this exact Dockerfile can serve a request before approval. The shared benchmark seed would also be useful to include in the same update. Validation scope: static review of the PR, shared chart, and relevant pinned upstream implementations. I did not build the image or run the recipes on a GB300/GKE cluster. |
ca71ede to
b16ec7e
Compare
|
Thanks @syangx39, addressed in b16ec7e:
|
b16ec7e to
b775628
Compare
KV-aware vs. round-robin routing for coding-agent traffic, in its own nemotron3-ultra-kv-vs-rr/ folder under both A4X-MAX Dynamo recipes: - disaggregated-serving/dynamo/nemotron3-ultra-kv-vs-rr/: 8P8D (16 nodes / 64 GPUs, one NVL72 ComputeDomain, mooncake KV transfer over NVLink). - aggregated-serving/dynamo/nemotron3-ultra-kv-vs-rr/: 6 workers (6 nodes / 24 GPUs). - Each folder has a dedicated README (why routing matters, files, architecture, deploying both arms, A/B benchmark, router tuning, troubleshooting) and paired values files with identical workers: *_kv.yaml (tuned KV-aware router) and *_rr.yaml (round-robin baseline). - Serving configs under src/frameworks/a4xmax/dynamo-configs. - arm64 serving image (SGLang CUDA 13 + ai-dynamo 1.4.2 + FlashInfer 0.6.18) under src/docker/dynamo-sglang-n3u. - Short pointer sections in both parent READMEs and top-level index entries.
b775628 to
4a00257
Compare
KV-aware vs. round-robin routing for AgentX agentic workload