Skip to content

fix(gpt-oss): give each scanned GPT-OSS layer its own vLLM KV cache - #6027

Open
chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-gpt-oss-scanned-vllm-kv-cache
Open

chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-gpt-oss-scanned-vllm-kv-cache

Conversation

@chiajunglien

@chiajunglien chiajunglien commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Description

vLLM serves MaxTextForCausalLM with one external KV cache per decoder layer and keeps the caches the decoder returns. With scan_layers=True, GPT-OSS scans a GptOssScannableBlock of two layers (sliding window, then global attention), and:

  • The generic scanned branch of NNXDecoder passed the flat cache list straight to _apply_layers_sequentially, which unrolls one iteration per block, so block i only got kv_caches[i]: half of the caches were never used, and the block handed one cache down both layers.
  • GptOssDecoderLayer returned None instead of the updated cache under scan_layers, like LlamaDecoderLayer (fix(llama): keep the vLLM KV cache in scanned Llama decoder layers #6002).

So the second layer of every block attended without a cache (forward_serve_vllm then returns the query), and vLLM decoded a scanned GPT-OSS checkpoint as gibberish from the first token, e.g. the RL checkpoint in Step 3 of test_gpt_oss_rl.sh. RL rollouts are not affected (they use the unscanned model).

Changes

  • NNXDecoder: for GPT-OSS (_SCANNED_BLOCKS_WITH_PER_LAYER_KV_CACHES), group the caches per block with prepare_kv_caches_for_scan and write them back with update_kv_caches_after_scan, as the Gemma3 and Qwen3-Next paths do.
  • GptOssScannableBlock: each layer gets its own kv_cache[layer_id]; returns the updated caches as a tuple (None without external caches).
  • GptOssDecoderLayer: returns its attention's cache under scan_layers too.
  • Other models are unchanged. The Llama4, Olmo3 and Envy blocks use the same generic branch and likely need the same change; not done here since it could not be verified end to end.

BUGS: b/579212908

Tests

Unit tests

CPU, 4 virtual devices: nnx_decoders_test.py and decoder_layer_model_mode_test.py: 167 passed, 2 skipped.

  • New TestNNXDecoderGptOssKvCaches: a 4-layer GPT-OSS decoder (two scanned blocks) gets one cache per layer; the attention is wrapped to bump the cache it returns, and every cache must come back bumped exactly once, scanned and unscanned.
  • It fails before this change (layer 0's KV cache update was dropped), and with only the GptOssDecoderLayer change (both layers of block 0 update one cache).

E2E on TPU v6e

v6e-32, Pathways (the maxtext_e2e_tpu_post_training setup), image tpu_post_training:047d606 + #6028 (GPT-OSS on the fused MoE path; it also adds the decode steps to the script), test_gpt_oss_rl.sh, checkpoint e2e-40609490996.

Step 3: decode the scanned RL checkpoint Output
Before duplication suggests we used when factor actually extending your adoptions not commonly used ...
After (this PR) analysisUser asks: "Suggest some famous landmarks in London." They want a list. ... Big Ben, Tower of London, Buckingham Palace, London Eye, ...

Both runs pass Step 1 (decode the unscanned converted checkpoint) and Step 2 (RL, GSM8K 81.25% → 95.0%).

Logs: before (Step 3 command at line 4096, output at 6143), after (4107, 6155). "Before" sampled at temperature 1.0, "after" is greedy. Both runs also applied #5988 (no effect with TP only).

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@chiajunglien chiajunglien added skip-ci-tests Add this label to the PR to skip running CI tests and removed skip-ci-tests Add this label to the PR to skip running CI tests labels Oct 29, 2026
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request fixes a critical issue where scanned GPT-OSS models produced incorrect output during vLLM serving. The root cause was improper handling of KV caches when scan_layers was enabled, where multiple layers shared a single cache or failed to update it. The changes ensure that each layer in a scanned block is correctly assigned its own KV cache and that updates are properly propagated back, restoring correct decoding behavior.

Highlights

  • KV Cache Management for Scanned GPT-OSS: Updated NNXDecoder to correctly group and update KV caches per layer when using scan_layers for GPT-OSS, ensuring each layer receives its own cache.
  • GptOssScannableBlock and Layer Updates: Modified GptOssScannableBlock to pass individual layer caches and return updated caches as a tuple, and updated GptOssDecoderLayer to correctly return attention caches under scan_layers.
  • Verification: Added a new unit test, TestNNXDecoderGptOssKvCaches, to verify that every GPT-OSS decoder layer receives and updates its own KV cache correctly.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: When serving a scanned GPT-OSS model with vLLM, decoding produces gibberish from the first token because the flat KV cache list is passed straight through the scan, starving the second layer of each block of its cache and dropping the cache updates.
Spec item Implementation Validating test
R1 (stated): Each layer in GptOssScannableBlock receives its own KV cache ✅ nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅
R2 (stated): GptOssDecoderLayer returns the updated cache when scanned ✅ nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅
R3 (stated): Updated caches are written back to the flat list for vLLM ✅ nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅
C1 (stated): Other models (e.g., Llama4, Olmo3) left unchanged as they are unverified ✅ preserved nnx_decoders_test.py::TestNNXDecoderGptOssKvCaches ✅

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.00000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/nnx_decoders.py 84.61% 0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

vLLM serves MaxTextForCausalLM with one external KV cache per decoder
layer and keeps the caches the decoder returns. With scan_layers=True,
GPT-OSS scans a GptOssScannableBlock of two layers (sliding window, then
global attention), but:

- The generic scanned branch of NNXDecoder passed the flat cache list
  straight to _apply_layers_sequentially, which unrolls one iteration per
  block, so block i only got kv_caches[i]. Half of the caches were never
  used, and the block handed that one cache down its two layers.
- GptOssDecoderLayer returned None instead of the updated cache under
  scan_layers, like LlamaDecoderLayer (#6002).

So the second layer of every block attended without a cache
(forward_serve_vllm returns the query as the attention output when it
gets none), and decoding a scanned GPT-OSS checkpoint with vLLM, e.g. the
RL checkpoint in test_gpt_oss_rl.sh, produced gibberish from the first
token.

For GPT-OSS, group the caches per block with prepare_kv_caches_for_scan
and write them back with update_kv_caches_after_scan, as the Gemma3 and
Qwen3-Next paths do; give each layer of GptOssScannableBlock its own
cache and return the updated caches as a tuple; and return the updated
cache from GptOssDecoderLayer under scan_layers too. Other models are
unchanged.

Bug: b/544614464
@chiajunglien
chiajunglien force-pushed the emma/fix-gpt-oss-scanned-vllm-kv-cache branch from 1535d37 to dcd656c Compare November 4, 2026 06:56
@chiajunglien chiajunglien removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants