Repository navigation
fix(vllm): return the vLLM KV cache from more scanned decoder layers - #6226
Open
chiajunglien wants to merge 1 commit into
Open
chiajunglien wants to merge 1 commit into
chiajunglien wants to merge 1 commit into
Conversation
When vLLM serves a scanned model, NNXDecoder statically unrolls the scanned layers, passes each layer its own kv_cache and keeps the cache the layer returns. #6002 fixed LlamaDecoderLayer, which returned None there under scan_layers and so dropped every cache update: the first token is right, every later one attends to a stale cache. The same pattern is in the Gemma, Gemma2, Mistral, Mixtral, Qwen2, Qwen3 custom MoE and GPT-3 decoder layers and in NNXDecoderLayer; return the cache there too. Outside vLLM, attention returns the kv_cache it was given (None), and the scan consumers only read the layer output, so other paths are unchanged. DecoderLayerScannedKvCacheTest now covers those layers along with Llama2, Qwen3 and Gemma3, instead of only Llama2 and Qwen3. DeepSeek's layers are left out: they do not hand the KV cache to their attention at all, scanned or not. The Llama3.1-70B RL and SFT scripts exited 0 whatever the decodes printed. They now save each decode's output and fail unless it mentions Big Ben, which greedy decoding of their prompt does. Also rewords the comment in llama2.py as suggested in the review of #6002. Bug: b/558524540, b/558524529
chiajunglien
requested review from
RissyRan,
bvandermoon,
gobbleturk,
jiangjy1982,
parambole and
shuningjin
as code owners
November 10, 2026 03:55
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
chiajunglien
requested review from
A9isha,
NuojCheng,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
darisoy,
dipannita08,
gagika,
hengtaoguo,
igorts-git,
khatwanimohit,
shralex,
vipannalla and
xibinliu
as code owners
November 10, 2026 03:55
Contributor
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description; PR #6002; code before/after.
- Problem: When vLLM serves scanned models (
scan_layers=True), multiple single-layer decoders returnedNoneinstead of the attention KV cache, dropping cache updates after token 1 and corrupting decoding. Additionally, E2E test scripts exited 0 even when decode outputs were completely degraded.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Return (layer_output, kv_cache) from scanned single-layer decoders (Gemma, Gemma2, Mistral, Mixtral, Qwen2, Qwen3CustomMoe, Gpt3, NNXDecoderLayer) |
✅ met | tests/unit/decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTest ✅ |
R2 (stated): Expand DecoderLayerScannedKvCacheTest to cover all onboarded single-layer decoder classes (11 cases) under scan_layers=True |
✅ met | tests/unit/decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTest ✅ |
R3 (stated): Validate that test_llama3.1_70b_rl.sh and test_llama3.1_70b_sft.sh fail if decoding produces gibberish or fails (pipefail + output check for "Big Ben") |
✅ met | test_llama3.1_70b_rl.sh, test_llama3.1_70b_sft.sh ✅ |
C1 (stated): Training and non-vLLM inference paths remain unchanged when kv_cache=None or when the training scan discards intermediate KV caches |
✅ preserved | tests/unit/decoder_layer_model_mode_test.py ✅ |
C2 (inferred): Multi-layer scannable blocks (llama4, olmo3, envy) and non-vLLM layers (DeepSeek) are kept out of scope without regression |
✅ preserved | tests/unit/decoder_layer_model_mode_test.py ✅ |
SurbhiJainUSC
approved these changes
Nov 10, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #6002, from @A9isha's review.
When vLLM serves a scanned model,
NNXDecoderstatically unrolls the scanned layers, passes each layer its ownkv_cacheand keeps the cache the layer returns. #6002 fixedLlamaDecoderLayer, which returnedNonethere underscan_layersand so dropped every cache update (the first token is right, then gibberish). The same pattern is in other decoder layers.Changes
GemmaDecoderLayer,Gemma2DecoderLayer,MistralDecoderLayer,MixtralDecoderLayer,Qwen2DecoderLayer,Qwen3CustomMoeDecoderLayer,Gpt3DecoderLayerandNNXDecoderLayerreturn the attention'skv_cacheunderscan_layerstoo, as fix(llama): keep the vLLM KV cache in scanned Llama decoder layers #6002 did for Llama. Outside vLLM, attention returns thekv_cacheit was given (None), and the scan consumers only read the layer output, so other paths are unchanged.DecoderLayerScannedKvCacheTestnow covers those layers along with Llama2, Qwen3 and Gemma3 (11 cases, was Llama2 and Qwen3).test_llama3.1_70b_rl.shandtest_llama3.1_70b_sft.shexited 0 whatever the decodes printed. They now setpipefail, save each decode's output withtee, and fail unless it mentions "Big Ben", which greedy decoding of their prompt does (see the outputs in fix(llama): keep the vLLM KV cache in scanned Llama decoder layers #6002).llama2.pyreads as suggested in the review.Not in this PR
llama4,olmo3,envy: their scanned module is a block of several layers, which also has to give each layer its own cache, as fix(gpt-oss): give each scanned GPT-OSS layer its own vLLM KV cache #6027 does for GPT-OSS. I'll send that after fix(gpt-oss): give each scanned GPT-OSS layer its own vLLM KV cache #6027 merges, and add GPT-OSS (fixed by fix(gpt-oss): give each scanned GPT-OSS layer its own vLLM KV cache #6027) to the test then.DeepSeekDenseLayerandDeepSeekMoELayernever hand the KV cache to their attention (attention_opdoes not take it, andMLAhas no vLLM path), scanned or not, so this fix does not apply. They are left out of the test.BUGS: b/558524540, b/558524529
Tests
CPU:
decoder_layer_model_mode_test: 47 passed.nnx_decoders_test: 134 passed, 2 skipped.DecoderLayerScannedKvCacheTestagainst main'ssrc/: the 8 changed layers fail (None is not Array(...)), and Llama2, Qwen3 and Gemma3 pass.pipefail.No TPU run: the script decodes are unchanged, and #6002 shows their greedy outputs (
Big Ben, Buckingham Palace, ...at steps 1 and 3).Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.