Repository navigation
fix(gpt-oss): give each scanned GPT-OSS layer its own vLLM KV cache - #6027
chiajunglien wants to merge 1 commit into
Conversation
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request fixes a critical issue where scanned GPT-OSS models produced incorrect output during vLLM serving. The root cause was improper handling of KV caches when scan_layers was enabled, where multiple layers shared a single cache or failed to update it. The changes ensure that each layer in a scanned block is correctly assigned its own KV cache and that updates are properly propagated back, restoring correct decoding behavior. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: When serving a scanned GPT-OSS model with vLLM, decoding produces gibberish from the first token because the flat KV cache list is passed straight through the scan, starving the second layer of each block of its cache and dropping the cache updates.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Each layer in GptOssScannableBlock receives its own KV cache |
✅ | nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅ |
R2 (stated): GptOssDecoderLayer returns the updated cache when scanned |
✅ | nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅ |
| R3 (stated): Updated caches are written back to the flat list for vLLM | ✅ | nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅ |
| C1 (stated): Other models (e.g., Llama4, Olmo3) left unchanged as they are unverified | ✅ preserved | nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅ |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
vLLM serves MaxTextForCausalLM with one external KV cache per decoder layer and keeps the caches the decoder returns. With scan_layers=True, GPT-OSS scans a GptOssScannableBlock of two layers (sliding window, then global attention), but: - The generic scanned branch of NNXDecoder passed the flat cache list straight to _apply_layers_sequentially, which unrolls one iteration per block, so block i only got kv_caches[i]. Half of the caches were never used, and the block handed that one cache down its two layers. - GptOssDecoderLayer returned None instead of the updated cache under scan_layers, like LlamaDecoderLayer (#6002). So the second layer of every block attended without a cache (forward_serve_vllm returns the query as the attention output when it gets none), and decoding a scanned GPT-OSS checkpoint with vLLM, e.g. the RL checkpoint in test_gpt_oss_rl.sh, produced gibberish from the first token. For GPT-OSS, group the caches per block with prepare_kv_caches_for_scan and write them back with update_kv_caches_after_scan, as the Gemma3 and Qwen3-Next paths do; give each layer of GptOssScannableBlock its own cache and return the updated caches as a tuple; and return the updated cache from GptOssDecoderLayer under scan_layers too. Other models are unchanged. Bug: b/544614464
1535d37 to
dcd656c
Compare
Description
vLLM serves
MaxTextForCausalLMwith one external KV cache per decoder layer and keeps the caches the decoder returns. Withscan_layers=True, GPT-OSS scans aGptOssScannableBlockof two layers (sliding window, then global attention), and:NNXDecoderpassed the flat cache list straight to_apply_layers_sequentially, which unrolls one iteration per block, so blockionly gotkv_caches[i]: half of the caches were never used, and the block handed one cache down both layers.GptOssDecoderLayerreturnedNoneinstead of the updated cache underscan_layers, likeLlamaDecoderLayer(fix(llama): keep the vLLM KV cache in scanned Llama decoder layers #6002).So the second layer of every block attended without a cache (
forward_serve_vllmthen returns the query), and vLLM decoded a scanned GPT-OSS checkpoint as gibberish from the first token, e.g. the RL checkpoint in Step 3 oftest_gpt_oss_rl.sh. RL rollouts are not affected (they use the unscanned model).Changes
NNXDecoder: for GPT-OSS (_SCANNED_BLOCKS_WITH_PER_LAYER_KV_CACHES), group the caches per block withprepare_kv_caches_for_scanand write them back withupdate_kv_caches_after_scan, as the Gemma3 and Qwen3-Next paths do.GptOssScannableBlock: each layer gets its ownkv_cache[layer_id]; returns the updated caches as a tuple (Nonewithout external caches).GptOssDecoderLayer: returns its attention's cache underscan_layerstoo.BUGS: b/579212908
Tests
Unit tests
CPU, 4 virtual devices:
nnx_decoders_test.pyanddecoder_layer_model_mode_test.py: 167 passed, 2 skipped.TestNNXDecoderGptOssKvCaches: a 4-layer GPT-OSS decoder (two scanned blocks) gets one cache per layer; the attention is wrapped to bump the cache it returns, and every cache must come back bumped exactly once, scanned and unscanned.layer 0's KV cache update was dropped), and with only theGptOssDecoderLayerchange (both layers of block 0 update one cache).E2E on TPU v6e
v6e-32, Pathways (the
maxtext_e2e_tpu_post_trainingsetup), imagetpu_post_training:047d606+ #6028 (GPT-OSS on the fused MoE path; it also adds the decode steps to the script),test_gpt_oss_rl.sh, checkpointe2e-40609490996.duplication suggests we used when factor actually extending your adoptions not commonly used ...analysisUser asks: "Suggest some famous landmarks in London." They want a list. ... Big Ben, Tower of London, Buckingham Palace, London Eye, ...Both runs pass Step 1 (decode the unscanned converted checkpoint) and Step 2 (RL, GSM8K 81.25% → 95.0%).
Logs: before (Step 3 command at line 4096, output at 6143), after (4107, 6155). "Before" sampled at temperature 1.0, "after" is greedy. Both runs also applied #5988 (no effect with TP only).
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.