Skip to content

fix(vllm): return the vLLM KV cache from more scanned decoder layers - #6226

Open
chiajunglien wants to merge 1 commit into
mainfrom
emma/return-kv-cache-from-scanned-decoder-layers
Open

chiajunglien wants to merge 1 commit into
mainfrom
emma/return-kv-cache-from-scanned-decoder-layers

Conversation

@chiajunglien

Copy link
Copy Markdown
Collaborator

Description

Follow-up to #6002, from @A9isha's review.

When vLLM serves a scanned model, NNXDecoder statically unrolls the scanned layers, passes each layer its own kv_cache and keeps the cache the layer returns. #6002 fixed LlamaDecoderLayer, which returned None there under scan_layers and so dropped every cache update (the first token is right, then gibberish). The same pattern is in other decoder layers.

Changes

  • GemmaDecoderLayer, Gemma2DecoderLayer, MistralDecoderLayer, MixtralDecoderLayer, Qwen2DecoderLayer, Qwen3CustomMoeDecoderLayer, Gpt3DecoderLayer and NNXDecoderLayer return the attention's kv_cache under scan_layers too, as fix(llama): keep the vLLM KV cache in scanned Llama decoder layers #6002 did for Llama. Outside vLLM, attention returns the kv_cache it was given (None), and the scan consumers only read the layer output, so other paths are unchanged.
  • DecoderLayerScannedKvCacheTest now covers those layers along with Llama2, Qwen3 and Gemma3 (11 cases, was Llama2 and Qwen3).
  • test_llama3.1_70b_rl.sh and test_llama3.1_70b_sft.sh exited 0 whatever the decodes printed. They now set pipefail, save each decode's output with tee, and fail unless it mentions "Big Ben", which greedy decoding of their prompt does (see the outputs in fix(llama): keep the vLLM KV cache in scanned Llama decoder layers #6002).
  • The comment in llama2.py reads as suggested in the review.

Not in this PR

BUGS: b/558524540, b/558524529

Tests

CPU: decoder_layer_model_mode_test: 47 passed. nnx_decoders_test: 134 passed, 2 skipped.

  • DecoderLayerScannedKvCacheTest against main's src/: the 8 changed layers fail (None is not Array(...)), and Llama2, Qwen3 and Gemma3 pass.
  • The output check in the scripts, simulated locally: a decode that mentions Big Ben passes, gibberish exits 1, and a failing decode exits with its own code through pipefail.

No TPU run: the script decodes are unchanged, and #6002 shows their greedy outputs (Big Ben, Buckingham Palace, ... at steps 1 and 3).

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

When vLLM serves a scanned model, NNXDecoder statically unrolls the
scanned layers, passes each layer its own kv_cache and keeps the cache
the layer returns. #6002 fixed LlamaDecoderLayer, which returned None
there under scan_layers and so dropped every cache update: the first
token is right, every later one attends to a stale cache. The same
pattern is in the Gemma, Gemma2, Mistral, Mixtral, Qwen2, Qwen3 custom
MoE and GPT-3 decoder layers and in NNXDecoderLayer; return the cache
there too. Outside vLLM, attention returns the kv_cache it was given
(None), and the scan consumers only read the layer output, so other
paths are unchanged.

DecoderLayerScannedKvCacheTest now covers those layers along with
Llama2, Qwen3 and Gemma3, instead of only Llama2 and Qwen3. DeepSeek's
layers are left out: they do not hand the KV cache to their attention
at all, scanned or not.

The Llama3.1-70B RL and SFT scripts exited 0 whatever the decodes
printed. They now save each decode's output and fail unless it mentions
Big Ben, which greedy decoding of their prompt does. Also rewords the
comment in llama2.py as suggested in the review of #6002.

Bug: b/558524540, b/558524529
@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description; PR #6002; code before/after.
  • Problem: When vLLM serves scanned models (scan_layers=True), multiple single-layer decoders returned None instead of the attention KV cache, dropping cache updates after token 1 and corrupting decoding. Additionally, E2E test scripts exited 0 even when decode outputs were completely degraded.
Spec item Implementation Validating test
R1 (stated): Return (layer_output, kv_cache) from scanned single-layer decoders (Gemma, Gemma2, Mistral, Mixtral, Qwen2, Qwen3CustomMoe, Gpt3, NNXDecoderLayer) ✅ met tests/unit/decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTest ✅
R2 (stated): Expand DecoderLayerScannedKvCacheTest to cover all onboarded single-layer decoder classes (11 cases) under scan_layers=True ✅ met tests/unit/decoder_layer_model_mode_test.py::DecoderLayerScannedKvCacheTest ✅
R3 (stated): Validate that test_llama3.1_70b_rl.sh and test_llama3.1_70b_sft.sh fail if decoding produces gibberish or fails (pipefail + output check for "Big Ben") ✅ met test_llama3.1_70b_rl.sh, test_llama3.1_70b_sft.sh ✅
C1 (stated): Training and non-vLLM inference paths remain unchanged when kv_cache=None or when the training scan discards intermediate KV caches ✅ preserved tests/unit/decoder_layer_model_mode_test.py ✅
C2 (inferred): Multi-layer scannable blocks (llama4, olmo3, envy) and non-vLLM layers (DeepSeek) are kept out of scope without regression ✅ preserved tests/unit/decoder_layer_model_mode_test.py ✅

@chiajunglien chiajunglien removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants