Skip to content

fix(llama): keep the vLLM KV cache in scanned Llama decoder layers - #6002

Closed
chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-llama31-vllm-decode-greedy
Closed

chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-llama31-vllm-decode-greedy

Conversation

@chiajunglien

@chiajunglien chiajunglien commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The Llama3.1-70B vLLM decodes in test_llama3.1_70b_rl.sh and test_llama3.1_70b_sft.sh print gibberish while the scripts still exit 0 (b/558524540, b/558524529). There are two independent causes.

1. Scanned Llama layers drop the vLLM KV cache (step 3, after SFT/RL)

LlamaDecoderLayer returned (layer_output, None) whenever scan_layers=True. When vLLM serves a scanned model, _apply_layers_sequentially statically unrolls the scanned layers, passes each layer its own kv_cache, and stores the cache the layer returns. So every KV cache update was dropped.

The first token is computed in the same attention call that writes the prompt's keys and values, so it is right. Every later token attends to a stale cache:

Big the the a” a the the the the ...

The step-3 decodes load the scanned post-training checkpoints with scan_layers=true, so they hit this bug. It is unrelated to SFT/RL: the scanned base checkpoint gives the same output.

Fix: return the KV cache under scan_layers too, as Qwen3DecoderLayer already does and as #4066 did for Gemma3. Outside vLLM, attention returns the kv_cache it was given (None), so other paths are unchanged.

2. Sampling at temperature 1.0 from the base model (step 1, before SFT/RL)

vllm_decode samples with decode_sampling_temperature, which defaults to 1.0. These tests decode the base meta-llama/Llama-3.1-70B checkpoint with the Instruct chat template. At temperature 1.0 that gives incoherent text (CopyWriter = ASSISTANT); greedy decoding of the same prompt gives Big Ben, Buckingham Palace, Tower Bridge, ....

Fix: pass decode_sampling_temperature=0.0 in the scripts, as the usage example in vllm_decode.py does.

Bugs: b/558524540, b/558524529


Tests

Unit tests

tests/unit/decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTest runs on CPU. It checks that a scanned decoder layer returns the KV cache its attention returns, for Llama2 and Qwen3.

  • Before the fix: fails for Llama2 (None is not Array(...)).
  • After the fix: passes.
  • Whole file: 34 passed.

E2E on TPU v6e-32

  • Image: tpu_post_training:1ed249c, with this change applied for the "after" runs.
  • Runtime: Pathways.
  • Model: Llama3.1-70B, checkpoint e2e-39297931486.
  • Prompt: "Suggest some famous landmarks in London.", with the Instruct chat template.
Checkpoint Decode Before After
Base, unscanned (step 1) TP8, temperature 1.0 → 0.0 CopyWriter = ASSISTANT Big Ben, Buckingham Palace, Tower Bridge, London Eye, ...
After SFT, scanned (step 3) TP16, scan_layers=true, greedy 1 the the a” a the the the ... 1. Big Ben: Big Ben is one of the most famous landmarks in London. It is a clock tower that is located in the Palace of Westminster. ...
After RL, scanned (step 3) TP16, scan_layers=true, greedy Big the the a” a the the the ... Big Ben, Buckingham Palace, Tower Bridge, London Eye, Trafalgar Square, ...

Isolation runs on the base checkpoint, greedy, TP16, without the fix:

  • Unscanned (scan_layers=false): Big Ben, Buckingham Palace, .... This rules out the KV-head replication (8 → 16) at TP16.
  • Scanned (scan_layers=true): Big the the a” a the the .... This shows the failure comes from the scanned path, not from training.

Logs

  • Step 1 at temperature 1.0, before the fix (from the full RL script run): log (decode command at line 1037, output at line 3017)
  • Base checkpoint, greedy, TP8: log (outputs at lines 1994, 3960 and 5936; the last one uses the chat template)
  • After SFT and after RL, greedy, before the fix: log (outputs at lines 2015 and 4013)
  • Isolation, unscanned vs scanned at TP16, before the fix: log (outputs at lines 2320 and 4299)
  • After SFT and after RL, greedy, with the fix: log (outputs at lines 2020 and 4013)

E2E re-run after the rebase (2026-11-04)

Checkpoint Decode Without this PR With this PR
Base, unscanned (step 1) TP8 CopyWriter = ASSISTANT (temperature 1.0, as in the current script; from RC run e2e-41748547965) Big Ben, Buckingham Palace, Tower Bridge, London Eye, Trafalgar Square, Westminster Abbey, ...
After SFT, scanned (step 3) TP16, scan_layers=true Here the the , and he (temperature 1.0, as in the current script; log) 1. Big Ben: Big Ben is one of the most famous landmarks in London. It is a clock tower that is located in the Palace of Westminster. ...

Log with this PR: log (step 1 output at line 2139, step 3 output at line 4203).

Notes

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses issues causing incoherent text generation in Llama3.1-70B vLLM decodes. The changes ensure that the KV cache is correctly preserved when using scanned layers and enforce a greedy decoding strategy (temperature 0.0) in test scripts to improve output quality. These fixes resolve reported bugs related to stale cache states and non-deterministic sampling.

Highlights

  • KV Cache Preservation: Updated LlamaDecoderLayer to correctly return the KV cache when scan_layers is enabled, preventing cache updates from being dropped during vLLM serving.
  • Inference Configuration: Updated end-to-end test scripts to use a decoding temperature of 0.0, ensuring deterministic and coherent output during base model evaluation.
  • Regression Testing: Added a new unit test, DecoderLayerScannedKvCacheTest, to verify that scanned decoder layers properly propagate KV cache updates.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met; the fix correctly preserves the KV cache in Llama2 and corrects the test script sampling temperature.

Top issues:

  1. 🟠 [Bug] The exact bug fixed here (returning layer_output, None when scan_layers=True outside of is_scan_carry) exists in almost all other models (gemma.py, gemma2.py, deepseek.py, mistral.py, mixtral.py, qwen2.py, gpt_oss.py, olmo3.py, envy.py). → Consider fixing them in this PR or opening a follow-up issue so they don't silently fail when served with vLLM under scan_layers=True.
Inferred spec & validation
  • Sources: PR description; linked Buganizer tickets (b/558524540, b/558524529, not readable here); code before/after.
  • Problem: Scanned Llama decoder layers drop the vLLM KV cache, and test scripts use a high sampling temperature on the base model, both causing gibberish text in vLLM decodes.
Spec item Implementation Validating test
R1 (stated): LlamaDecoderLayer with scan_layers=True must return the updated KV cache from attention. ✅ met decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTest ✅
R2 (stated): test_llama3.1_70b_rl.sh and test_llama3.1_70b_sft.sh must use decode_sampling_temperature=0.0. ✅ met ❌ none
C1 (inferred): Unscanned Llama layers and non-vLLM behavior must remain unchanged. ✅ preserved decoder_layer_model_mode_test.py ✅

@codecov

codecov Bot commented Oct 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@chiajunglien
chiajunglien force-pushed the emma/fix-llama31-vllm-decode-greedy branch from 3182063 to 21f4808 Compare November 4, 2026 04:07
chiajunglien added a commit that referenced this pull request Nov 4, 2026
vLLM serves MaxTextForCausalLM with one external KV cache per decoder
layer and keeps the caches the decoder returns. With scan_layers=True,
GPT-OSS scans a GptOssScannableBlock of two layers (sliding window, then
global attention), but:

- The generic scanned branch of NNXDecoder passed the flat cache list
  straight to _apply_layers_sequentially, which unrolls one iteration per
  block, so block i only got kv_caches[i]. Half of the caches were never
  used, and the block handed that one cache down its two layers.
- GptOssDecoderLayer returned None instead of the updated cache under
  scan_layers, like LlamaDecoderLayer (#6002).

So the second layer of every block attended without a cache
(forward_serve_vllm returns the query as the attention output when it
gets none), and decoding a scanned GPT-OSS checkpoint with vLLM, e.g. the
RL checkpoint in test_gpt_oss_rl.sh, produced gibberish from the first
token.

For GPT-OSS, group the caches per block with prepare_kv_caches_for_scan
and write them back with update_kv_caches_after_scan, as the Gemma3 and
Qwen3-Next paths do; give each layer of GptOssScannableBlock its own
cache and return the updated caches as a tuple; and return the updated
cache from GptOssDecoderLayer under scan_layers too. Other models are
unchanged.

Bug: b/544614464

@A9isha A9isha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — clear diagnosis and thorough E2E evidence. I verified the change is a no-op outside vLLM: every scan/pipeline/prefetch consumer reads only layer_out[0], and non-vLLM Attention.__call__ returns the kv_cache it was given (None). The fix matches Qwen3DecoderLayer / Gemma3DecoderLayer.

Two requests (non-blocking):

  1. Same bug in other layers. I ran the new test parametrized over the file's existing _LAYERS list plus gemma3 and gpt_oss: deepseek_dense, deepseek_moe, gemma, gemma2, gpt_oss, mistral, qwen2 all fail with the same None is not Array(...). The elif cfg.scan_layers: return layer_output, None pattern is also in llama4.py, mixtral.py, envy.py, olmo3.py and nnx_decoders.py::NNXDecoderLayer. Since each fix is the same two lines and the test already supports those classes, could you fix them here and switch the test to *_LAYERS (+ gemma3/gpt_oss)? If you'd rather keep scope tight, please at least extend the test and mark the failing cases expectedFailure with a bug, so the gap is visible.

  2. Make the scripts fail on bad output. Both bugs persisted because the decodes exit 0 regardless of output. With greedy decoding now deterministic, a grep -q "Big Ben" on the decode output in the two scripts would have caught both. Fine as a follow-up.

Nit: the llama2.py comment "Also under scan_layers: …" reads as a fragment now that the elif is gone; suggest "Under scan_layers, vLLM statically unrolls the layers and keeps the kv_cache returned here; the training scan discards it."

The Llama3.1-70B vLLM decodes in the RL and SFT E2E tests printed
gibberish for two independent reasons.

1. LlamaDecoderLayer returned (layer_output, None) whenever scan_layers
   was set. When vLLM serves a scanned model, the decoder statically
   unrolls the scanned layers, passes each layer its own kv_cache and
   keeps the cache each layer returns, so every KV cache update was
   dropped. The first token, computed in the attention call that also
   writes the prompt's keys and values, was right; every later token
   attended to a stale cache ("Big the the the ..."). This broke the
   step-3 decodes, which load the scanned post-training checkpoints.
   Return the KV cache under scan_layers as well, as Qwen3DecoderLayer
   already does and as #4066 did for Gemma3. Outside vLLM, attention
   returns the kv_cache it was given (None), so nothing else changes.

2. vllm_decode samples with decode_sampling_temperature (1.0 by
   default). The tests decode the base meta-llama/Llama-3.1-70B
   checkpoint with the Instruct chat template, which gives incoherent
   text at temperature 1.0. Decode greedily in the Llama3.1-70B RL and
   SFT scripts, as the vllm_decode usage example does.

Add a unit test that a scanned decoder layer returns the KV cache its
attention returns.

Bug: b/558524540
Bug: b/558524529
@chiajunglien
chiajunglien force-pushed the emma/fix-llama31-vllm-decode-greedy branch from 21f4808 to 8f74c1b Compare November 5, 2026 00:35
@chiajunglien chiajunglien reopened this Nov 5, 2026
@SurbhiJainUSC
SurbhiJainUSC self-requested a review November 5, 2026 00:58
copybara-service Bot pushed a commit that referenced this pull request Nov 5, 2026
COPYBARA_INTEGRATE_REVIEW=#6002 from AI-Hypercomputer:emma/fix-llama31-vllm-decode-greedy 21f4808
PiperOrigin-RevId: 1014251718
@chiajunglien

Copy link
Copy Markdown
Collaborator Author

LGTM — clear diagnosis and thorough E2E evidence. I verified the change is a no-op outside vLLM: every scan/pipeline/prefetch consumer reads only layer_out[0], and non-vLLM Attention.__call__ returns the kv_cache it was given (None). The fix matches Qwen3DecoderLayer / Gemma3DecoderLayer.

Two requests (non-blocking):

  1. Same bug in other layers. I ran the new test parametrized over the file's existing _LAYERS list plus gemma3 and gpt_oss: deepseek_dense, deepseek_moe, gemma, gemma2, gpt_oss, mistral, qwen2 all fail with the same None is not Array(...). The elif cfg.scan_layers: return layer_output, None pattern is also in llama4.py, mixtral.py, envy.py, olmo3.py and nnx_decoders.py::NNXDecoderLayer. Since each fix is the same two lines and the test already supports those classes, could you fix them here and switch the test to *_LAYERS (+ gemma3/gpt_oss)? If you'd rather keep scope tight, please at least extend the test and mark the failing cases expectedFailure with a bug, so the gap is visible.
  2. Make the scripts fail on bad output. Both bugs persisted because the decodes exit 0 regardless of output. With greedy decoding now deterministic, a grep -q "Big Ben" on the decode output in the two scripts would have caught both. Fine as a follow-up.

Nit: the llama2.py comment "Also under scan_layers: …" reads as a fragment now that the elif is gone; suggest "Under scan_layers, vLLM statically unrolls the layers and keeps the kv_cache returned here; the training scan discards it."

Thanks @A9isha for the thorough review, and for running the test against the other layers!

  • Same bug in other layers: I'll send a follow-up PR that applies the same fix to the other decoder layers (deepseek,
    gemma, gemma2, mistral, qwen2, llama4, mixtral, envy, olmo3 and NNXDecoderLayer) and extends the test to cover all of them. GPT-OSS is covered by fix(gpt-oss): give each scanned GPT-OSS layer its own vLLM KV cache #6027.
  • Failing on bad output: agreed. We're adding an output check to our E2E pipeline, and I'll also add a simple grep to these two scripts in the follow-up.
  • Nit: I'll fix the comment in the follow-up as well.

This change has already landed in main as dc7803a, so I'm closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants