Repository navigation
fix(llama): keep the vLLM KV cache in scanned Llama decoder layers - #6002
chiajunglien wants to merge 1 commit into
Conversation
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses issues causing incoherent text generation in Llama3.1-70B vLLM decodes. The changes ensure that the KV cache is correctly preserved when using scanned layers and enforce a greedy decoding strategy (temperature 0.0) in test scripts to improve output quality. These fixes resolve reported bugs related to stale cache states and non-deterministic sampling. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met; the fix correctly preserves the KV cache in Llama2 and corrects the test script sampling temperature.
Top issues:
- 🟠 [Bug] The exact bug fixed here (returning
layer_output, Nonewhenscan_layers=Trueoutside ofis_scan_carry) exists in almost all other models (gemma.py,gemma2.py,deepseek.py,mistral.py,mixtral.py,qwen2.py,gpt_oss.py,olmo3.py,envy.py). → Consider fixing them in this PR or opening a follow-up issue so they don't silently fail when served with vLLM underscan_layers=True.
Inferred spec & validation
- Sources: PR description; linked Buganizer tickets (b/558524540, b/558524529, not readable here); code before/after.
- Problem: Scanned Llama decoder layers drop the vLLM KV cache, and test scripts use a high sampling temperature on the base model, both causing gibberish text in vLLM decodes.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): LlamaDecoderLayer with scan_layers=True must return the updated KV cache from attention. |
✅ met | decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTest ✅ |
R2 (stated): test_llama3.1_70b_rl.sh and test_llama3.1_70b_sft.sh must use decode_sampling_temperature=0.0. |
✅ met | ❌ none |
| C1 (inferred): Unscanned Llama layers and non-vLLM behavior must remain unchanged. | ✅ preserved | decoder_layer_model_mode_test.py ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
3182063 to
21f4808
Compare
vLLM serves MaxTextForCausalLM with one external KV cache per decoder layer and keeps the caches the decoder returns. With scan_layers=True, GPT-OSS scans a GptOssScannableBlock of two layers (sliding window, then global attention), but: - The generic scanned branch of NNXDecoder passed the flat cache list straight to _apply_layers_sequentially, which unrolls one iteration per block, so block i only got kv_caches[i]. Half of the caches were never used, and the block handed that one cache down its two layers. - GptOssDecoderLayer returned None instead of the updated cache under scan_layers, like LlamaDecoderLayer (#6002). So the second layer of every block attended without a cache (forward_serve_vllm returns the query as the attention output when it gets none), and decoding a scanned GPT-OSS checkpoint with vLLM, e.g. the RL checkpoint in test_gpt_oss_rl.sh, produced gibberish from the first token. For GPT-OSS, group the caches per block with prepare_kv_caches_for_scan and write them back with update_kv_caches_after_scan, as the Gemma3 and Qwen3-Next paths do; give each layer of GptOssScannableBlock its own cache and return the updated caches as a tuple; and return the updated cache from GptOssDecoderLayer under scan_layers too. Other models are unchanged. Bug: b/544614464
A9isha
left a comment
There was a problem hiding this comment.
LGTM — clear diagnosis and thorough E2E evidence. I verified the change is a no-op outside vLLM: every scan/pipeline/prefetch consumer reads only layer_out[0], and non-vLLM Attention.__call__ returns the kv_cache it was given (None). The fix matches Qwen3DecoderLayer / Gemma3DecoderLayer.
Two requests (non-blocking):
-
Same bug in other layers. I ran the new test parametrized over the file's existing
_LAYERSlist plus gemma3 and gpt_oss:deepseek_dense,deepseek_moe,gemma,gemma2,gpt_oss,mistral,qwen2all fail with the sameNone is not Array(...). Theelif cfg.scan_layers: return layer_output, Nonepattern is also inllama4.py,mixtral.py,envy.py,olmo3.pyandnnx_decoders.py::NNXDecoderLayer. Since each fix is the same two lines and the test already supports those classes, could you fix them here and switch the test to*_LAYERS(+ gemma3/gpt_oss)? If you'd rather keep scope tight, please at least extend the test and mark the failing casesexpectedFailurewith a bug, so the gap is visible. -
Make the scripts fail on bad output. Both bugs persisted because the decodes exit 0 regardless of output. With greedy decoding now deterministic, a
grep -q "Big Ben"on the decode output in the two scripts would have caught both. Fine as a follow-up.
Nit: the llama2.py comment "Also under scan_layers: …" reads as a fragment now that the elif is gone; suggest "Under scan_layers, vLLM statically unrolls the layers and keeps the kv_cache returned here; the training scan discards it."
The Llama3.1-70B vLLM decodes in the RL and SFT E2E tests printed
gibberish for two independent reasons.
1. LlamaDecoderLayer returned (layer_output, None) whenever scan_layers
was set. When vLLM serves a scanned model, the decoder statically
unrolls the scanned layers, passes each layer its own kv_cache and
keeps the cache each layer returns, so every KV cache update was
dropped. The first token, computed in the attention call that also
writes the prompt's keys and values, was right; every later token
attended to a stale cache ("Big the the the ..."). This broke the
step-3 decodes, which load the scanned post-training checkpoints.
Return the KV cache under scan_layers as well, as Qwen3DecoderLayer
already does and as #4066 did for Gemma3. Outside vLLM, attention
returns the kv_cache it was given (None), so nothing else changes.
2. vllm_decode samples with decode_sampling_temperature (1.0 by
default). The tests decode the base meta-llama/Llama-3.1-70B
checkpoint with the Instruct chat template, which gives incoherent
text at temperature 1.0. Decode greedily in the Llama3.1-70B RL and
SFT scripts, as the vllm_decode usage example does.
Add a unit test that a scanned decoder layer returns the KV cache its
attention returns.
Bug: b/558524540
Bug: b/558524529
21f4808 to
8f74c1b
Compare
Thanks @A9isha for the thorough review, and for running the test against the other layers!
This change has already landed in main as dc7803a, so I'm closing this PR. |
Description
The Llama3.1-70B vLLM decodes in
test_llama3.1_70b_rl.shandtest_llama3.1_70b_sft.shprint gibberish while the scripts still exit 0 (b/558524540, b/558524529). There are two independent causes.1. Scanned Llama layers drop the vLLM KV cache (step 3, after SFT/RL)
LlamaDecoderLayerreturned(layer_output, None)wheneverscan_layers=True. When vLLM serves a scanned model,_apply_layers_sequentiallystatically unrolls the scanned layers, passes each layer its ownkv_cache, and stores the cache the layer returns. So every KV cache update was dropped.The first token is computed in the same attention call that writes the prompt's keys and values, so it is right. Every later token attends to a stale cache:
The step-3 decodes load the scanned post-training checkpoints with
scan_layers=true, so they hit this bug. It is unrelated to SFT/RL: the scanned base checkpoint gives the same output.Fix: return the KV cache under
scan_layerstoo, asQwen3DecoderLayeralready does and as #4066 did for Gemma3. Outside vLLM, attention returns thekv_cacheit was given (None), so other paths are unchanged.2. Sampling at temperature 1.0 from the base model (step 1, before SFT/RL)
vllm_decodesamples withdecode_sampling_temperature, which defaults to 1.0. These tests decode the basemeta-llama/Llama-3.1-70Bcheckpoint with the Instruct chat template. At temperature 1.0 that gives incoherent text (CopyWriter = ASSISTANT); greedy decoding of the same prompt givesBig Ben, Buckingham Palace, Tower Bridge, ....Fix: pass
decode_sampling_temperature=0.0in the scripts, as the usage example invllm_decode.pydoes.Bugs: b/558524540, b/558524529
Tests
Unit tests
tests/unit/decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTestruns on CPU. It checks that a scanned decoder layer returns the KV cache its attention returns, for Llama2 and Qwen3.None is not Array(...)).E2E on TPU v6e-32
tpu_post_training:1ed249c, with this change applied for the "after" runs.e2e-39297931486.CopyWriter = ASSISTANTBig Ben, Buckingham Palace, Tower Bridge, London Eye, ...scan_layers=true, greedy1 the the a” a the the the ...1. Big Ben: Big Ben is one of the most famous landmarks in London. It is a clock tower that is located in the Palace of Westminster. ...scan_layers=true, greedyBig the the a” a the the the ...Big Ben, Buckingham Palace, Tower Bridge, London Eye, Trafalgar Square, ...Isolation runs on the base checkpoint, greedy, TP16, without the fix:
scan_layers=false):Big Ben, Buckingham Palace, .... This rules out the KV-head replication (8 → 16) at TP16.scan_layers=true):Big the the a” a the the .... This shows the failure comes from the scanned path, not from training.Logs
E2E re-run after the rebase (2026-11-04)
tpu_post_training:41748547965(MaxTexte71398e, 2026-11-03), with this change applied tollama2.py.e2e-41748547965. Step 3 useshbm_utilization_vllm=0.5from fix(testing): lower the HBM utilization of the Llama3.1-70B post-training vLLM decodes #6128.CopyWriter = ASSISTANT(temperature 1.0, as in the current script; from RC rune2e-41748547965)Big Ben, Buckingham Palace, Tower Bridge, London Eye, Trafalgar Square, Westminster Abbey, ...scan_layers=trueHere the the , and he(temperature 1.0, as in the current script; log)1. Big Ben: Big Ben is one of the most famous landmarks in London. It is a clock tower that is located in the Palace of Westminster. ...Log with this PR: log (step 1 output at line 2139, step 3 output at line 4203).
Notes
main, which now has the v6e settings from feat(testing): adapt the E2E pre-/post-training scripts to TPU v6e #5990 (TP16 for the scanned step-3 decodes). This PR only addsdecode_sampling_temperature=0.0to the scripts.hbm_utilization_vllm0.6 → 0.5) to load. The two PRs change different lines and merge cleanly in either order.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.