Skip to content

fix(qwen3): route vLLM per-layer attention metadata through scanned hybrid blocks - #5933

Merged
copybara-service[bot] merged 1 commit into
mainfrom
emma/fix-qwen35-scanned-vllm-metadata
Oct 23, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
emma/fix-qwen35-scanned-vllm-metadata

Conversation

@chiajunglien

Copy link
Copy Markdown
Collaborator

Description

Since #5009, Qwen3.5 reuses Qwen3NextScannableBlock, which raises NotImplementedError: Per-layer attention_metadata (a dict keyed by layer) requires scan_layers=False. when vLLM passes per-layer attention metadata (a dict keyed by layer.{i}, one entry per KV-cache group, which vLLM sends for hybrid GDN + full-attention models). The sub-layers of a scanned block do not know their global layer index, so the block cannot resolve the dict itself. vllm_decode forwards scan_layers to the vLLM adapter, so vllm_decode ... scan_layers=True (Steps 1 and 3 of tests/end_to_end/tpu/qwen3.5/35b/test_qwen3.5_rl.sh) has failed on the first forward pass since #5009 merged.

DAG Error Log

With external (vLLM) KV caches the decoder already unrolls the blocks statically, so this change resolves the dict per block in the decoder instead:

  • NNXDecoder._apply_qwen3_hybrid_scanned_blocks looks each layer up by its global index (the same lookup as the unscanned path, now shared as layer_attention_metadata) and hands every block, and the remainder block, a tuple with its sub-layers' entries, ordered like the block's KV caches.
  • _apply_layers_sequentially takes per_iteration_kwargs (static-unroll path only), passed as call arguments so they stay correct under jax.checkpoint.
  • Qwen3NextScannableBlock accepts that per-sub-layer tuple together with external KV caches, and still rejects an unresolved dict.

The training path (no external KV caches) is unchanged.

Tests

  • New unit tests in tests/unit/nnx_decoders_test.py:
    • Block level: each sub-layer receives its own entry; an unresolved dict, a per-sub-layer tuple without external caches, and a tuple of the wrong length are rejected.
    • Decoder level (Qwen3-Next and Qwen3.5): every layer of a two-block + remainder model receives the entry of its global index; a per-layer dict without KV caches is still rejected.
    • The new decoder test fails on main with the same NotImplementedError as the E2E test and passes with this change.
  • python3 -m pytest -q tests/unit/nnx_decoders_test.py tests/unit/grouped_attention_metadata_test.py: 132 passed, 2 skipped.
  • pyink and codespell are clean; pylint reports no new messages (only the ones already present on main).
  • E2E on v5p-64 (Pathways, submitted with gcluster using the same flags as the maxtext_e2e_tpu_post_training DAG): Step 1 of test_qwen3.5_rl.sh (vllm_decode with scan_layers=True on the scanned checkpoint) now completes and the script moves on to Step 2

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a regression in Qwen3.5 models where per-layer attention metadata, provided by vLLM, could not be correctly resolved within scanned blocks. Because scanned sub-layers lack knowledge of their global layer index, the decoder now resolves the global metadata dictionary into a per-sub-layer tuple before passing it to the blocks. This ensures compatibility with hybrid GDN + full-attention models while maintaining support for existing training paths.

Highlights

  • Attention Metadata Routing: Implemented a mechanism to resolve per-layer attention metadata for scanned hybrid blocks in Qwen3/3.5 models, ensuring each sub-layer receives the correct metadata entry corresponding to its global layer index.
  • Decoder Integration: Updated NNXDecoder to handle per-iteration keyword arguments, allowing static unrolling of blocks to correctly pass specific attention metadata to sub-layers when using external vLLM KV caches.
  • Validation and Testing: Added comprehensive unit tests to verify metadata routing at both block and decoder levels, and added validation to reject invalid metadata configurations (e.g., unresolved dicts in scanned blocks).
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

…ybrid blocks

Since Qwen3.5 reuses Qwen3NextScannableBlock, a scanned Qwen3-Next/Qwen3.5
decoder raises NotImplementedError when vLLM hands it per-layer attention
metadata (a dict keyed by `layer.{i}`, one entry per KV-cache group), because
the sub-layers of a scanned block do not know their global layer index.
vllm_decode forwards `scan_layers` to the adapter, so
`vllm_decode ... scan_layers=True` -- used by the Qwen3.5 RL E2E test on its
scanned checkpoints -- fails on the first forward pass.

With external (vLLM) KV caches the decoder already unrolls the blocks
statically, so it now resolves the dict per block instead:
- NNXDecoder._apply_qwen3_hybrid_scanned_blocks looks each layer up by its
  global index (the same lookup as the unscanned path) and hands every block,
  and the remainder block, a tuple with its sub-layers' entries, ordered like
  the block's KV caches.
- _apply_layers_sequentially takes `per_iteration_kwargs` (static-unroll path
  only), passed as call arguments so they survive jax.checkpoint.
- Qwen3NextScannableBlock accepts that per-sub-layer tuple together with
  external KV caches, and still rejects an unresolved dict.
@chiajunglien
chiajunglien force-pushed the emma/fix-qwen35-scanned-vllm-metadata branch from 79f01a1 to 3d53ba2 Compare October 23, 2026 08:57

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found. Spec fully met with robust test coverage.

Inferred spec & validation
  • Sources: PR description; linked PR #5009; tests/end_to_end/tpu/qwen3.5/35b/test_qwen3.5_rl.sh; code before/after.
  • Problem: vllm_decode with scan_layers=True on Qwen3 hybrid models (Qwen3-Next and Qwen3.5) raised NotImplementedError because sub-layers of a scanned block do not know their global layer index and cannot resolve vLLM's per-layer attention metadata dict.
Spec item Implementation Validating test
R1 (stated): When external KV caches are supplied, NNXDecoder._apply_qwen3_hybrid_scanned_blocks resolves the per-layer attention_metadata dict into per-block tuples by global layer index for both scanned blocks and the remainder block. ✅ tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_routes_per_layer_attention_metadata_by_global_layer ✅
R2 (stated): _apply_layers_sequentially threads per_iteration_kwargs into each statically unrolled layer call as call arguments rather than captured closures. ✅ tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_routes_per_layer_attention_metadata_by_global_layer ✅
R3 (stated): Qwen3NextScannableBlock unpacks the per-sub-layer metadata tuple in block order, matching each sub-layer's KV cache. ✅ tests/unit/nnx_decoders_test.py::Qwen3_5ScannableBlockKVCacheTest::test_each_sublayer_gets_its_own_attention_metadata ✅
R4 (stated): A per-layer attention_metadata dict without external KV caches is rejected by Qwen3NextScannableBlock. ✅ tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_rejects_per_layer_attention_metadata_without_kv_caches ✅
R5 (stated): A per-sub-layer attention_metadata tuple passed without external KV caches, or with an invalid length, is rejected. ✅ tests/unit/nnx_decoders_test.py::Qwen3_5ScannableBlockKVCacheTest::test_rejects_per_sub_layer_attention_metadata_without_external_caches ✅, test_rejects_per_sub_layer_attention_metadata_of_the_wrong_length ✅
C1 (stated): Training path (no external KV caches) behavior and performance remain unchanged. ✅ preserved tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_rejects_per_layer_attention_metadata_without_kv_caches ✅
C2 (inferred): Unscanned execution (scan_layers=False) routes per-layer attention metadata without regressions. ✅ preserved tests/unit/grouped_attention_metadata_test.py::GroupedAttentionMetadataRoutingTest::test_nnx_decoder_unscanned_routes_grouped_metadata_per_layer ✅

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.00000% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/nnx_decoders.py 83.33% 2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@copybara-service
copybara-service Bot merged commit 9cceb08 into main Oct 23, 2026
78 of 79 checks passed
@copybara-service
copybara-service Bot deleted the emma/fix-qwen35-scanned-vllm-metadata branch October 23, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants