Repository navigation
fix(testing): lower the HBM utilization of the Llama3.1-70B post-training vLLM decodes - #6128
Conversation
…ning vLLM decodes The post-SFT and post-RL vLLM decodes in test_llama3.1_70b_sft.sh and test_llama3.1_70b_rl.sh (TP16, scan_layers=true) set hbm_utilization_vllm=0.6. On v6e that leaves 12.04 GiB per chip after the weights (10.87 GiB) and the KV cache, and loading the compiled jit_run_model_impl program now needs 12.83 GiB, so both decodes fail with RESOURCE_EXHAUSTED (RuntimeProgramAllocationFailure). Lower hbm_utilization_vllm to 0.5 for these two decodes. The KV cache shrinks from 3411 to 2100 blocks, which is still far more than a single 2048-token prompt needs, and the program fits with about 2.4 GiB to spare.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a memory allocation failure occurring during Llama3.1-70B post-training E2E tests on v6e hardware. Due to an increase in the memory footprint of the compiled model program, the previous HBM utilization setting was causing runtime exhaustion. By lowering the utilization threshold, the KV cache size is reduced, allowing the workload to execute successfully while maintaining sufficient capacity for inference. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: The Llama3.1-70B post-training E2E tests (SFT and RL) fail in their final vLLM decode step on v6e with
RESOURCE_EXHAUSTED. The compiledjit_run_model_implprogram size grew, sohbm_utilization_vllm=0.6no longer leaves enough memory to load it.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Lower hbm_utilization_vllm to 0.5 for the post-SFT and post-RL Llama 3.1 70B decodes on v6e to prevent OOM. |
✅ | test_llama3.1_70b_sft.sh ✅ |
| C1 (stated): The KV cache remains large enough for the prompt. | ✅ preserved | test_llama3.1_70b_sft.sh ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Description
The Llama3.1-70B post-training E2E tests fail in their last step on v6e. The post-SFT and post-RL vLLM decodes (
ici_tensor_parallelism=16,scan_layers=true) usehbm_utilization_vllm=0.6, which leaves 12.04 GiB per chip after the bf16 weights (10.87 GiB) and the KV cache. Loading the compiledjit_run_model_implprogram now needs 12.83 GiB:In the RC run e2e-41748547965 (2026-11-04), this failed all four attempts of the Llama3.1-70B SFT test and the Llama3.1-70B RL attempts, each time in this decode. The same settings passed on 2026-10-22 with the same weight and KV cache sizes, so the program's own buffers grew since then. The Pathways server build (
pathways/server:latest, 2026-10-18 build then, 2026-11-03 build now) and the MaxText code both changed in between.This PR lowers
hbm_utilization_vllmto 0.5 for these two decodes. The KV cache shrinks from 3411 to 2100 blocks, which is still far more than a single 2048-token prompt needs, and the program fits with about 2.4 GiB to spare.Tests
Re-ran only the post-SFT decode of
test_llama3.1_70b_sft.shon v6e-32 (4x8) with Pathways, with the same image (tpu_post_training:41748547965), the same Pathways server build (2026-11-03, pinned by digest), and the same SFT checkpoint as the failing run:hbm_utilization_vllmRESOURCE_EXHAUSTEDas above, in all four attempts of the RC runThe post-RL decode in
test_llama3.1_70b_rl.shuses the same TP16 layout, so each chip holds the same weights (10.87 GiB) and loads the same program. In the RC run it failed with the same error.The decoded text is gibberish with or without this change. That is a separate bug, fixed in #6002.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.