Repository navigation
fix(qwen3): route vLLM per-layer attention metadata through scanned hybrid blocks - #5933
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a regression in Qwen3.5 models where per-layer attention metadata, provided by vLLM, could not be correctly resolved within scanned blocks. Because scanned sub-layers lack knowledge of their global layer index, the decoder now resolves the global metadata dictionary into a per-sub-layer tuple before passing it to the blocks. This ensures compatibility with hybrid GDN + full-attention models while maintaining support for existing training paths. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
…ybrid blocks
Since Qwen3.5 reuses Qwen3NextScannableBlock, a scanned Qwen3-Next/Qwen3.5
decoder raises NotImplementedError when vLLM hands it per-layer attention
metadata (a dict keyed by `layer.{i}`, one entry per KV-cache group), because
the sub-layers of a scanned block do not know their global layer index.
vllm_decode forwards `scan_layers` to the adapter, so
`vllm_decode ... scan_layers=True` -- used by the Qwen3.5 RL E2E test on its
scanned checkpoints -- fails on the first forward pass.
With external (vLLM) KV caches the decoder already unrolls the blocks
statically, so it now resolves the dict per block instead:
- NNXDecoder._apply_qwen3_hybrid_scanned_blocks looks each layer up by its
global index (the same lookup as the unscanned path) and hands every block,
and the remainder block, a tuple with its sub-layers' entries, ordered like
the block's KV caches.
- _apply_layers_sequentially takes `per_iteration_kwargs` (static-unroll path
only), passed as call arguments so they survive jax.checkpoint.
- Qwen3NextScannableBlock accepts that per-sub-layer tuple together with
external KV caches, and still rejects an unresolved dict.
79f01a1 to
3d53ba2
Compare
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found. Spec fully met with robust test coverage.
Inferred spec & validation
- Sources: PR description; linked PR #5009;
tests/end_to_end/tpu/qwen3.5/35b/test_qwen3.5_rl.sh; code before/after. - Problem:
vllm_decodewithscan_layers=Trueon Qwen3 hybrid models (Qwen3-Next and Qwen3.5) raisedNotImplementedErrorbecause sub-layers of a scanned block do not know their global layer index and cannot resolve vLLM's per-layer attention metadata dict.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): When external KV caches are supplied, NNXDecoder._apply_qwen3_hybrid_scanned_blocks resolves the per-layer attention_metadata dict into per-block tuples by global layer index for both scanned blocks and the remainder block. |
✅ | tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_routes_per_layer_attention_metadata_by_global_layer ✅ |
R2 (stated): _apply_layers_sequentially threads per_iteration_kwargs into each statically unrolled layer call as call arguments rather than captured closures. |
✅ | tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_routes_per_layer_attention_metadata_by_global_layer ✅ |
R3 (stated): Qwen3NextScannableBlock unpacks the per-sub-layer metadata tuple in block order, matching each sub-layer's KV cache. |
✅ | tests/unit/nnx_decoders_test.py::Qwen3_5ScannableBlockKVCacheTest::test_each_sublayer_gets_its_own_attention_metadata ✅ |
R4 (stated): A per-layer attention_metadata dict without external KV caches is rejected by Qwen3NextScannableBlock. |
✅ | tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_rejects_per_layer_attention_metadata_without_kv_caches ✅ |
R5 (stated): A per-sub-layer attention_metadata tuple passed without external KV caches, or with an invalid length, is rejected. |
✅ | tests/unit/nnx_decoders_test.py::Qwen3_5ScannableBlockKVCacheTest::test_rejects_per_sub_layer_attention_metadata_without_external_caches ✅, test_rejects_per_sub_layer_attention_metadata_of_the_wrong_length ✅ |
| C1 (stated): Training path (no external KV caches) behavior and performance remain unchanged. | ✅ preserved | tests/unit/nnx_decoders_test.py::TestNNXDecoderQwen3Next::test_decoder_rejects_per_layer_attention_metadata_without_kv_caches ✅ |
C2 (inferred): Unscanned execution (scan_layers=False) routes per-layer attention metadata without regressions. |
✅ preserved | tests/unit/grouped_attention_metadata_test.py::GroupedAttentionMetadataRoutingTest::test_nnx_decoder_unscanned_routes_grouped_metadata_per_layer ✅ |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Description
Since #5009, Qwen3.5 reuses
Qwen3NextScannableBlock, which raisesNotImplementedError: Per-layer attention_metadata (a dict keyed by layer) requires scan_layers=False.when vLLM passes per-layer attention metadata (a dict keyed bylayer.{i}, one entry per KV-cache group, which vLLM sends for hybrid GDN + full-attention models). The sub-layers of a scanned block do not know their global layer index, so the block cannot resolve the dict itself.vllm_decodeforwardsscan_layersto the vLLM adapter, sovllm_decode ... scan_layers=True(Steps 1 and 3 oftests/end_to_end/tpu/qwen3.5/35b/test_qwen3.5_rl.sh) has failed on the first forward pass since #5009 merged.DAG Error Log
With external (vLLM) KV caches the decoder already unrolls the blocks statically, so this change resolves the dict per block in the decoder instead:
NNXDecoder._apply_qwen3_hybrid_scanned_blockslooks each layer up by its global index (the same lookup as the unscanned path, now shared aslayer_attention_metadata) and hands every block, and the remainder block, a tuple with its sub-layers' entries, ordered like the block's KV caches._apply_layers_sequentiallytakesper_iteration_kwargs(static-unroll path only), passed as call arguments so they stay correct underjax.checkpoint.Qwen3NextScannableBlockaccepts that per-sub-layer tuple together with external KV caches, and still rejects an unresolved dict.The training path (no external KV caches) is unchanged.
Tests
tests/unit/nnx_decoders_test.py:mainwith the sameNotImplementedErroras the E2E test and passes with this change.python3 -m pytest -q tests/unit/nnx_decoders_test.py tests/unit/grouped_attention_metadata_test.py: 132 passed, 2 skipped.pyinkandcodespellare clean;pylintreports no new messages (only the ones already present onmain).gclusterusing the same flags as themaxtext_e2e_tpu_post_trainingDAG): Step 1 of test_qwen3.5_rl.sh (vllm_decode with scan_layers=True on the scanned checkpoint) now completes and the script moves on to Step 2Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.