Skip to content

fix(testing): lower the HBM utilization of the Llama3.1-70B post-training vLLM decodes - #6128

Merged
copybara-service[bot] merged 1 commit into
mainfrom
emma/llama31-70b-decode-hbm
Nov 4, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
emma/llama31-70b-decode-hbm

Conversation

@chiajunglien

Copy link
Copy Markdown
Collaborator

Description

The Llama3.1-70B post-training E2E tests fail in their last step on v6e. The post-SFT and post-RL vLLM decodes (ici_tensor_parallelism=16, scan_layers=true) use hbm_utilization_vllm=0.6, which leaves 12.04 GiB per chip after the bf16 weights (10.87 GiB) and the KV cache. Loading the compiled jit_run_model_impl program now needs 12.83 GiB:

jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Execute event error: E0101: RuntimeProgramAllocationFailure:
Error loading program 'jit_run_model_impl': Attempting to reserve 12.83G at the bottom of memory. That was not possible. There are 12.04G free, 0B reserved, and 12.04G reservable.

In the RC run e2e-41748547965 (2026-11-04), this failed all four attempts of the Llama3.1-70B SFT test and the Llama3.1-70B RL attempts, each time in this decode. The same settings passed on 2026-10-22 with the same weight and KV cache sizes, so the program's own buffers grew since then. The Pathways server build (pathways/server:latest, 2026-10-18 build then, 2026-11-03 build now) and the MaxText code both changed in between.

This PR lowers hbm_utilization_vllm to 0.5 for these two decodes. The KV cache shrinks from 3411 to 2100 blocks, which is still far more than a single 2048-token prompt needs, and the program fits with about 2.4 GiB to spare.

Tests

Re-ran only the post-SFT decode of test_llama3.1_70b_sft.sh on v6e-32 (4x8) with Pathways, with the same image (tpu_post_training:41748547965), the same Pathways server build (2026-11-03, pinned by digest), and the same SFT checkpoint as the failing run:

hbm_utilization_vllm KV cache Result
0.6 (before) 3411 blocks RESOURCE_EXHAUSTED as above, in all four attempts of the RC run
0.5 (after) 2100 blocks Decode completes, exit 0 (log)

The post-RL decode in test_llama3.1_70b_rl.sh uses the same TP16 layout, so each chip holds the same weights (10.87 GiB) and loads the same program. In the RC run it failed with the same error.

The decoded text is gibberish with or without this change. That is a separate bug, fixed in #6002.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

…ning vLLM decodes

The post-SFT and post-RL vLLM decodes in test_llama3.1_70b_sft.sh and
test_llama3.1_70b_rl.sh (TP16, scan_layers=true) set
hbm_utilization_vllm=0.6. On v6e that leaves 12.04 GiB per chip after the
weights (10.87 GiB) and the KV cache, and loading the compiled
jit_run_model_impl program now needs 12.83 GiB, so both decodes fail with
RESOURCE_EXHAUSTED (RuntimeProgramAllocationFailure).

Lower hbm_utilization_vllm to 0.5 for these two decodes. The KV cache
shrinks from 3411 to 2100 blocks, which is still far more than a single
2048-token prompt needs, and the program fits with about 2.4 GiB to spare.
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a memory allocation failure occurring during Llama3.1-70B post-training E2E tests on v6e hardware. Due to an increase in the memory footprint of the compiled model program, the previous HBM utilization setting was causing runtime exhaustion. By lowering the utilization threshold, the KV cache size is reduced, allowing the workload to execute successfully while maintaining sufficient capacity for inference.

Highlights

  • HBM Utilization Adjustment: Reduced the hbm_utilization_vllm parameter from 0.6 to 0.5 in Llama3.1-70B post-training E2E tests to resolve memory exhaustion errors.
  • Resource Management: Adjusted the KV cache allocation to ensure the model fits within the available HBM on v6e hardware, successfully preventing RESOURCE_EXHAUSTED runtime errors.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: The Llama3.1-70B post-training E2E tests (SFT and RL) fail in their final vLLM decode step on v6e with RESOURCE_EXHAUSTED. The compiled jit_run_model_impl program size grew, so hbm_utilization_vllm=0.6 no longer leaves enough memory to load it.
Spec item Implementation Validating test
R1 (stated): Lower hbm_utilization_vllm to 0.5 for the post-SFT and post-RL Llama 3.1 70B decodes on v6e to prevent OOM. ✅ test_llama3.1_70b_sft.sh ✅
C1 (stated): The KV cache remains large enough for the prompt. ✅ preserved test_llama3.1_70b_sft.sh ✅

@codecov

codecov Bot commented Nov 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@chiajunglien chiajunglien removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 4, 2026
@copybara-service
copybara-service Bot merged commit 68678f2 into main Nov 4, 2026
111 of 112 checks passed
@copybara-service
copybara-service Bot deleted the emma/llama31-70b-decode-hbm branch November 4, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants