Skip to content

Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan - #5009

Merged
copybara-service[bot] merged 12 commits into
mainfrom
chengnuojin-qwen35-scan
Oct 21, 2026
Merged

copybara-service[bot] merged 12 commits into
mainfrom
chengnuojin-qwen35-scan

Conversation

@NuojCheng

@NuojCheng NuojCheng commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator

What

Extends the Qwen3-Next scanned-block work from #4964 to Qwen3.5.

Qwen3.5 repeats the exact same hybrid attention period as Qwen3-Next — inhomogeneous_layer_cycle_interval = 4, i.e. three GatedDeltaNet layers followed by one full-attention layer — so rather than porting the logic, this makes Qwen3.5 reuse it.

The two architectures now share the decoder layer as well as the block. Qwen3_5DecoderLayer subclasses Qwen3NextDecoderLayer and overrides only three factory hooks — _make_full_attention, _make_linear_attention, _make_mlp — so the forward pass, both LayerNorms, the residual structure, the is_full_attention_layer derivation and the MoE load-balance handling are inherited rather than duplicated. The hooks keep the submodule attribute names (input_layernorm, attention, post_attention_layernorm, mlp), so the checkpoint parameter paths are identical either way. Qwen3_5ScannableBlock likewise subclasses Qwen3NextScannableBlock and overrides only _make_decoder_layer.

models/qwen3_5.py goes from 266 to 127 lines. Qwen3.5 picks up the inner scan over the homogeneous linear-attention layers, the trip-count-one lax.scan scheduling barrier around the full-attention layer, and per-sub-layer remat, for free.

Changes

  • models/qwen3.py — extract sub-block construction from Qwen3NextDecoderLayer.__init__ into _make_full_attention() / _make_linear_attention() / _make_mlp(), and layer construction from Qwen3NextScannableBlock.__init__ into _make_decoder_layer(). These are the only points a sibling architecture has to diverge at. Also threads forced_routed_experts through the layer and the block (see below). No behavior change for Qwen3-Next.
  • models/qwen3_5.py — Qwen3_5DecoderLayer and Qwen3_5ScannableBlock become thin subclasses that override the hooks. Qwen3NextDecoderLayer gains an explicit is_full_attention_layer kwarg, because inside a scan a sub-layer's position in the cycle is not recoverable from layer_idx; it still falls back to the (layer_idx + 1) % cycle == 0 derivation when unscanned. The MoE load-balance loss is assigned rather than sow-n — sow appends to a tuple, so the layer's Intermediate structure grows on every call, which a lax.scan body cannot express.
  • layers/decoders.py, layers/nnx_decoders.py — route QWEN3_5 through the shared hybrid path. _init_scanned_qwen3_next / _apply_qwen3_next_scanned_blocks are renamed to ..._qwen3_hybrid_... and pick the block class by decoder_block, including for layers_remainder. forced_routed_experts_per_layer() is factored out of reshape_forced_routed_experts_for_scan() so the hybrid path can slice routing per sub-layer.
  • layers/nnx_scan.py — apply_scanned_layers() gains an optional xs pytree of per-layer scan inputs. Callers that pass no xs keep the two-argument apply_fn signature; Gemma4, the only other caller, is unchanged.
  • layers/moe.py — hold the FSDP expert-kernel all-gather inside the decoder loop when weight_dtype == dtype. Not part of the port; it is a pre-existing bug in the scanned MoE path that this PR would otherwise widen to Qwen3.5. See The weight_dtype=bfloat16 blow-up below.
  • checkpoint_conversion/utils/param_mapping.py — remap Qwen3.5's scanned checkpoint mapping from the old per-cycle-position layers-layer_{i} prefixes to the local_layers / global_layer tree, mirroring what perf(qwen3-next): nested scan over hybrid attention with per-layer remat #4964 did for Qwen3-Next. Local layers nest [block][local], the global layer is flat [block].
  • checkpoint_conversion/utils/tensor_handling.py — stacked_axes() was guarded on isinstance(mt_key, str), so Qwen3.5's composite (wi_0, wi_1) key (fed by one fused HF gate_up_proj tensor) fell through to the leading-axis MoE layout instead of the nested-scan layout and would have silently transposed those weights. Tuple keys now resolve via their first component.
  • integration/vllm/weight_converter.py, integration/vllm/maxtext_vllm_rollout.py — teach the RL rollout weight sync to read the nested tree (see below).

Results

AOT temp_size_in_bytes per device, using the config from train_compile_test.py::test_qwen3_5 (per_device_batch_size=1, max_target_length=1024, sparse_matmul, megablox, attention=flash, use_tokamax_splash, remat_policy=full — the base.yml default — on both sides). Both sides freshly compiled against main @ 457040f7:

model topology before after delta
qwen3.5-35b-a3b tpu7x-16 30.36 GB 25.13 GB −17.2% ✅
qwen3.5-35b-a3b tpu7x-8 43.61 GB 44.11 GB +1.1%
qwen3.5-397b-a17b v5p-512 75.62 GB 46.01 GB −39.2% ✅
qwen3.5-397b-a17b v5p-128 128.05 GB (OOM) 116.25 GB (OOM) −9%

Exact bytes: 35b/tpu7x-16 30359507040 → 25132249440; 35b/tpu7x-8 43612869024 → 44113347456; 397b/v5p-512 75622737696 → 46011566848. The 397b/v5p-128 OOM row is the original pre-rebase measurement and was not re-run.

The saving is a roughly constant 5–7 GB, not a constant percentage. Sweeping device count on identical silicon (qwen3.5-35b-a3b, tpu7x-N):

devices before after delta
8 43.61 GB 44.11 GB +0.50 GB
16 30.36 GB 25.13 GB −5.23 GB
32 21.50 GB 15.42 GB −6.08 GB
64 19.81 GB 14.70 GB −5.11 GB
128 15.59 GB 8.51 GB −7.08 GB

What shrinks with device count is the baseline, not the saving. The FSDP-sharded share of the working set falls as 1/N, while what this PR removes does not shard at all: sparse_matmul deliberately strips fsdp from the expert-kernel pspecs, so the gathered per-device MoE kernels are full size on every chip, and block-level remat keeps all four sub-layers' worth of them co-live. Per-sub-layer remat keeps one. The same absolute saving is therefore 17% of a 30 GB budget and 45% of a 16 GB one.

The cross-generation control agrees: v5p-128 and tpu7x-64 are both 64 JAX devices (v5p slice names count TensorCores) and start from quite different baselines — 16.16 GB and 19.81 GB — yet save 5.129 GB and 5.105 GB, within 0.5% of each other. v5p-512 (256 devices) saves 5.25 GB, 11.99 → 6.74 GB.

argument_size_in_bytes is unchanged in both directions (26.01 GB at 35b/tpu7x-16, 18.61 GB at 397b/v5p-512). Generated code size roughly halves on every row, including the ones where temp memory does not move — that is the nested scan replacing an unrolled cycle, and it is topology-independent: 35b/tpu7x-8 205 → 99 MB, 35b/tpu7x-16 210 → 99 MB, 397b/v5p-512 320 → 163 MB.

Real training on a v7-8 (qwen3.5-35b-a3b, synthetic data, ici_fsdp_parallelism=8, per_device_batch_size=1, max_target_length=1024, 6 steps) converges on both sides with matching loss curves (step 5: 7.056 before, 7.072 after — RNG-fork differences only). Memory there is a wash, 40.6 → 41.1 GB estimated temp, consistent with the tpu7x-8 row.

The layer-sharing refactor is free. Recompiling qwen3.5-35b-a3b at tpu7x-16 after collapsing Qwen3_5DecoderLayer onto Qwen3NextDecoderLayer and threading router replay gives temp_size_in_bytes=25132249440, generated_code_size_in_bytes=99265536 — byte-identical to the build before the refactor.

Test plan

All CPU-only, run individually against this branch:

suite result
tests/unit/nnx_decoders_test.py 65 passed, 2 skipped (-k Qwen3: 17 passed, up from 7 on main)
tests/unit/moe_test.py 38 passed, 40 skipped (TPU-only)
tests/unit/router_replay_test.py 38 passed, 1 skipped
tests/unit/nnx_scan_test.py 4 passed
tests/unit/param_mapping_test.py 30 passed
tests/post_training/unit/weight_converter_test.py 27 passed
  • The three Qwen3-Next test classes in nnx_decoders_test.py are parameterized by class attributes so the Qwen3.5 subclasses reuse them: nested-scan output matches a sequential unroll numerically, and the Linen and NNX parameter trees match, both for a whole number of periods and with a remainder. New: a block given forced routing that does not cover every sub-layer must raise.
  • moe_test.py adds test_sparse_matmul_pins_expert_kernel_all_gather_when_the_upcast_is_an_identity, parameterized over (bf16, bf16), (f32, f32) and (f32, bf16). It asserts the barrier is applied exactly when the upcast is an identity, and — by object identity, since the stand-in kernels share a shape — that the sharding constraint consumes the barrier's outputs rather than the raw kernels, so a barrier whose results are dropped does not pass. Both matching-dtype cases fail without the moe.py change.
  • param_mapping_test.py covers the local_layers / global_layer mapping shape and the composite-key axis fix.
  • router_replay_test.py adds a case for the scan remainder block, where only the trailing layer's routing differs between two runs, so the losses can diverge only if the remainder replays its own slice. Reverting the qwen3.py / nnx_scan.py / nnx_decoders.py changes fails test_loss_fn_with_forced_routed_experts_scanned_qwen3_5 and test_per_layer_routing_is_not_broadcast_scanned.
  • weight_converter_test.py adds a nested-layout source tree and asserts it converts byte-identically to the per-slot layout, plus a truncated cycle-slot axis being rejected. Both fail if the weight_converter.py change is reverted.
  • tests/post_training/unit/vllm_rollout_unroll_test.py adds a nested local_layers + global_layer unroll case and a missing-slot-axis rejection. This file needs vllm installed to import the module under test, so it was verified in CI-equivalent form by executing the unroll helpers in isolation: the nested layout produces a dense layers_{0..7} with every slot on its own layer, and the legacy per-slot and homogeneous paths are unchanged. The same check fails against main.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements hierarchical nested scans for Qwen3-Next and Qwen3.5 hybrid decoders, enabling per-layer rematerialization to optimize memory usage. It introduces scannable blocks that structure inhomogeneous attention cycles and updates checkpoint conversion utilities to support multi-axis stacked tensors with arbitrary nesting depths. Comprehensive unit tests are added to verify decoder parity and state restoration. The review feedback suggests updating the type annotations for mt_key from str to str | tuple[str, ...] in stacked_axes and _build_multi_axis_stacked_tensor to accurately reflect that composite keys can be represented as tuples.

return depth


def stacked_axes(mt_key: str, config, depth: int) -> tuple:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The mt_key parameter is annotated as str, but the function explicitly handles tuple inputs (lines 69-70). To ensure type safety and prevent static analysis warnings, the type annotation should be updated to str | tuple[str, ...].

Suggested change
def stacked_axes(mt_key: str, config, depth: int) -> tuple:
def stacked_axes(mt_key: str | tuple[str, ...], config, depth: int) -> tuple:

tensor_getter_fn: Callable[[str], np.ndarray],
hook_fns: Any,
target_leaf: Any,
config,
mt_key: str = "",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Update the type annotation of mt_key to support tuple inputs as well, aligning with the updated stacked_axes signature.

Suggested change
mt_key: str = "",
mt_key: str | tuple[str, ...] = "",

@@ -344,66 +344,59 @@ def get_maxtext_model_info(config):


def _build_multi_axis_stacked_tensor(
hf_source_keys: List[List[str]],
hf_source_keys: List[Any],
tensor_getter_fn: Callable[[str], np.ndarray],
hook_fns: Any,
target_shape: tuple,
config,
mt_key: str = "",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Update the type annotation of mt_key to support tuple inputs as well, aligning with the updated stacked_axes signature.

Suggested change
mt_key: str = "",
mt_key: str | tuple[str, ...] = "",

@codecov

codecov Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

Base automatically changed from chengnuojin-bharatgen-scan to main September 2, 2026 17:18
@NuojCheng
NuojCheng force-pushed the chengnuojin-qwen35-scan branch from ec280c7 to 8f902ef Compare September 3, 2026 05:48
@NuojCheng NuojCheng changed the title perf(qwen3.5): reuse Qwen3-Next's nested block scan Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan Sep 4, 2026
@NuojCheng
NuojCheng force-pushed the chengnuojin-qwen35-scan branch 2 times, most recently from 682b3d7 to 2892a98 Compare October 13, 2026 06:06
@NuojCheng
NuojCheng force-pushed the chengnuojin-qwen35-scan branch from 2892a98 to ee342ff Compare October 19, 2026 20:11
@NuojCheng
NuojCheng marked this pull request as ready for review October 19, 2026 20:11


class Qwen3_5ScannableBlock(nnx.Module):
class Qwen3_5ScannableBlock(Qwen3NextScannableBlock):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scanned Qwen3.5 checkpoints change layout: layers/layer_{slot} becomes layers/local_layers + layers/global_layer. Old scanned checkpoints won't load or convert. Can you note this in the PR description with a migration path, or keep a fallback for the old keys?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, scan-format changes gonna impact previous checkpoints. The ideal way is to re-run the to_maxtext.py to reproduce checkpoints.

prefix = f"params-decoder-layers-layer_{block_idx}"
# The length-1 _scan_global_layer scan is a memory boundary, not a param stack, so
# global_layer params stay flat [block] while local_layers nest [block][local].
num_blocks = num_main_layers // layer_cycle_interval

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the layer count isn't a multiple of the cycle, the model now builds layers_remainder, but nothing maps it. Those layers fail on MaxText→HF and are skipped on HF→MaxText. Can you add a mapping for layers_remainder-local_layers-*, or raise a clear error? Same at L1565 for Qwen3-Next.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Comment thread src/maxtext/models/qwen3.py Resolved
@@ -2684,19 +2718,22 @@ def _apply_qwen3_next_scanned_blocks(self, y, layer_args, layer_kwargs, kv_cache
length=scan_length,
kv_caches_stacked=grouped_kv_caches,
skip_block_remat=True,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This passes skip_block_remat=True, but with prefetch_fsdp_weights=True, _apply_layers_sequentially hands off to the prefetch path (L1207) without it. Qwen3.5 blocks remat their own sub-layers, so they get rematerialized twice. Can you forward skip_block_remat there?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

def test_per_layer_routing_reaches_the_scan_remainder_block(self):
"""Layers past the last whole cycle live in a separate remainder block."""
seq_len, batch_size, top_k, num_experts = 8, 1, 2, 4
cycle_interval, layers = 2, 3

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With cycle = 2 there's only one local layer per block, so these tests can't catch routing that's sent to the wrong sub-layer. Can you add a case with cycle >= 3 and a remainder, checking that scanned and unscanned losses match? Covering Qwen3-Next too would be nice.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

model_mode=self.model_mode,
quant=self.quant,
lambda layer_rngs: self._make_decoder_layer(
layer_idx=0,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Every local layer gets layer_idx=0, so dict-based attention_metadata maps them all to "layer.0". Probably fine since vLLM runs unscanned, but worth a check.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

Comment thread src/maxtext/models/qwen3.py Outdated
if self.global_layer is not None:
global_kv = kv_cache[self.num_local] if (kv_cache is not None and self.num_local < len(kv_cache)) else None
y, new_kv = self._run_layer(self.global_layer, y, layer_kwargs, global_kv)
global_routing = None if forced_routed_experts is None else forced_routed_experts[self.num_local]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This looks unreachable: the decoder raises when forced routing and external KV caches are combined. Add a comment or remove it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

Comment thread src/maxtext/models/qwen3.py Outdated
)
mlp_output, load_balance_loss = self._apply_mlp(hidden_states, deterministic, forced_routed_experts)

# We sow the load balancing loss so it can be collected and added to the total loss

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "We sow the load balancing loss" and "Assigned, not sown" contradict each other. Merge into one comment?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merged into one

param_scan_axis: int,
apply_fn: Callable[[nnx.Module, Any], Any],
apply_fn: Callable[..., Any],
xs: Any | None = None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Please document the new xs argument in the docstring. "Gemma4 is currently the only caller" is also out of date.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

mt_key = mt_key[0]
if isinstance(mt_key, str) and "-local_layers" in mt_key:
param_scan_axis = config.param_scan_axis
nested_axes = (param_scan_axis, param_scan_axis + 1)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: With param_scan_axis=0 and FP8 experts, this returns (0, 0, 1), and moveaxis fails on the repeated axis. Assert param_scan_axis >= 1, or handle 0.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It now raises ValueError

@NuojCheng
NuojCheng requested a review from jacoguzo as a code owner October 21, 2026 05:25

@darisoy darisoy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall.
biggest concern is around breaking the existing qwen3.5 checkpoints

NuojCheng and others added 12 commits October 21, 2026 17:41
Qwen3.5's decoder layer and scannable block were near-duplicates of
Qwen3-Next's. Collapse them onto the Qwen3-Next implementations, which
already nest a scan over the cycle's local layers inside a trip-count-one
scan around the full-attention layer, so Qwen3.5 picks up per-sub-layer
remat instead of block-level remat.

Sharing is via three overridable factory hooks on Qwen3NextDecoderLayer
(`_make_full_attention`, `_make_linear_attention`, `_make_mlp`). Qwen3.5
overrides only those, keeping the submodule attribute names -- and
therefore the checkpoint parameter paths -- identical, so the whole
forward pass, both norms and the residual structure are inherited rather
than copied. qwen3_5.py drops from 266 to 127 lines.

Two things had to be repaired to keep functionality identical:

Router replay. `_apply_layers_sequentially` filters call kwargs against
the callee signature, and Qwen3NextScannableBlock did not accept
`forced_routed_experts`, so scanned Qwen3.5 lost it. Thread it through
`nnx_scan.apply_scanned_layers` (new optional per-layer `xs`), the block
(scan xs for the local layers, closure for the global one, per-layer
slices on the external-KV unroll) and the hybrid scan entry point, which
also slices the scan remainder. Combining forced routing with
externally-managed KV caches now raises rather than silently dropping
one of them.

vLLM weight sync. The RL rollout converter and the direct-sync unroll
both assumed one module per cycle slot (`layers.layer_{slot}`) with a
single stacked axis. The nested block stores `local_layers` with the
cycle slot on `param_scan_axis + 1` and `global_layer` with the block
axis alone, so both now read that layout, selecting the slot once per
plan group. Trainer states in the old layout still convert. This also
fixes Qwen3-Next, which produced the nested layout already and was
therefore never syncable.

AOT compile, per_device_batch_size=1, max_target_length=1024, flash
attention with tokamax splash, temp_size_in_bytes vs main:

  qwen3.5-35b-a3b   v5p-128   16.16 GB -> 11.03 GB  (-31.7%)
  qwen3.5-35b-a3b   v5p-512   11.99 GB ->  6.74 GB  (-43.8%)
  qwen3.5-397b-a17b v5p-512   75.62 GB -> 46.01 GB  (-39.2%)
The routed-MoE kernel pspecs drop the fsdp axis so XLA emits the FSDP
all-gather at the sharding constraint in sparse_matmul. Under a scanned
decoder the kernels reaching that constraint are a per-layer dynamic-slice of
a stacked param, and with nothing in between XLA rewrites
all_gather(dynamic_slice(w)) into dynamic_slice(all_gather(w)). That gather is
loop invariant, so it is hoisted out of the scan and every layer's gathered
expert kernel is live at once -- num_layers * num_experts * embed * moe_mlp *
dtype bytes of extra peak memory.

The jnp.asarray(kernel, dtype) cast in __call__ normally separates the slice
from the collective and blocks the rewrite. It is an identity when
weight_dtype == dtype, which is why the blow-up only shows up there:
qwen3-next-80b-a3b at tpu7x-128 compiles to 14.20 GB of temporaries with
weight_dtype=float32 but 65.59 GB with weight_dtype=bfloat16, on code that is
byte-identical to main's.

Stand an optimization_barrier in for the cast in exactly that case, matching
the barrier moe_fsdp_use_two_stage_all_gather already uses a few lines above.
AOT temp_size_in_bytes per device:

  model                topology     before     after
  qwen3-next-80b-a3b   tpu7x-128    65.59 GB   12.11 GB
  qwen3.5-35b-a3b      tpu7x-128    30.23 GB    6.94 GB
  qwen3.5-35b-a3b      tpu7x-16     35.71 GB   17.04 GB
  qwen3-30b-a3b        tpu7x-16     13.03 GB   12.42 GB

Gating on the identity keeps every weight_dtype=float32 build byte-identical
(qwen3.5-35b-a3b 25132249440 at tpu7x-16 and 8510193056 at tpu7x-128,
qwen3-next-80b-a3b 14204451200 at tpu7x-128, before and after). Applying the
barrier unconditionally costs 3-8% on those.
pylint C3001 unnecessary-lambda-assignment on the no-router-replay branch of
Qwen3NextScannableBlock._scan_local_layers. It was the only finding failing
the Code Quality Check / Pre-commit Linters job. Mirrors the def already used
in the sibling branch; behaviour is unchanged.
Cut the explanatory comments and docstring prose added by this branch down to
the few that record something the code cannot show. 180 lines removed, 27
added; no code changes (AST identical, docstrings excluded).
`_scan_global_layer` carried the global-attention layer's params through the
scan carry and returned them unchanged, which makes them loop invariants of the
trip-count-one while loop. Recent XLA hoists the layer's FSDP expert weight
all-gathers out of that loop into the enclosing block scan, where they run
exposed before the local-layer scan and keep the gathered weights live across
it. Passing the params as scan xs (with a leading length-1 axis) keeps them
indexed by the loop, so each all-gather stays inside the global layer's loop.

Verified in TPU AoT (Qwen3.5-35B-A3B, gf=4x4x4, FSDP over all chips): with the
carry, the forward block body holds the three routed-expert all-gathers; with
xs, it holds none and the global layer's loop holds all of its all-gathers.
Compiled temp memory is unchanged.
`_scan_global_layer` feeds the global layer's params in as scan xs and drops
every `nnx.Param` on the way out, so a param materialized while tracing the
layer (e.g. a Qwix LoRA adapter or quantization param) never reached
`nnx.update` and was silently lost. Before params moved to xs they were in the
carry, so this would at least have failed with a structure mismatch.

Mirror `nnx_scan.apply_scanned_layers`: drop only the params that were fed in,
return the rest as an extra scan output, and apply them with `nnx.update`.
When no params are created that extra output is an empty pytree.

Adds a regression test (run for Qwen3-Next and Qwen3.5) that creates a param
inside the global layer's call and checks it lands on the block; it fails
without this change.
When num_decoder_layers is not a multiple of inhomogeneous_layer_cycle_interval,
the scanned decoder keeps the layers past the last whole cycle in a
`layers_remainder` block whose `local_layers` are stacked on param_scan_axis
alone. Nothing mapped those params, so MaxText -> HF failed on them and
HF -> MaxText skipped them: a 6-layer, cycle-4 Qwen3.5 or Qwen3-Next mapped
36 of its 53 params.

Map `layers_remainder-local_layers-*` (and register its hooks) for both models,
and stop `stacked_axes` from treating remainder keys as a nested block scan, so
remainder routed experts keep the plain (experts, layers) layout. Both models
now map 53/53 params at 6 layers, and still map exactly with no remainder keys
at 8.

`stacked_axes` also raises a clear error for expert-stacked weights in a nested
block scan with param_scan_axis=0, where the expert axis no longer leads and the
returned axes used to repeat, failing inside np.moveaxis. The type annotations
of `mt_key` now admit composite (tuple) keys.
`_apply_layers_sequentially` handed the prefetch path everything but
`skip_block_remat`. The prefetch pipeline rematerializes each scanned module, so
for a block that already rematerializes its own sub-layers (Qwen3-Next, Qwen3.5,
Gemma4, DeepSeek-V4) the backward pass recomputed the whole block and then, in
the block's own remat, every sub-layer again.

Simply dropping the pipeline's remat for such blocks is not an option: the
blocks' inner scans read their weights as scan xs (`moveaxis(w)` for
`local_layers`, `w[None]` for the global layer), so `jax.vjp` of the bare block
keeps them as residuals -- 35 residuals worth 101% of the block's weight bytes
for a Qwen3.5 block, i.e. every block's gathered weights saved between the
passes.

Instead, forward `skip_block_remat` and have the pipeline extend the remat
policy to also save each sub-layer's `decoder_layer_input`, which
Qwen3NextDecoderLayer now names like the other decoder layers do. The forward
then keeps the sub-layer inputs (2 residuals per Qwen3.5 block, no weights) and
the backward recomputes every sub-layer once. On the 8-layer CPU test config
with remat_policy=full, compiled value_and_grad FLOPs of the prefetch path drop
from 5.36e8 to 4.30e8: 4 forward-equivalents per block instead of 5, the same
as the regular scan. Loss and gradients match the non-prefetch path exactly for
Qwen3.5, Qwen3-Next and Gemma4.
MaxTextToMaxTextConverter:
- Take a `local_layers` source's cycle slot before casting it. Every slot's
  plan group reads the whole source, so casting first cast the full stack once
  per slot (3x the work at cycle 4).
- Require exactly `cycle - 1` slots on the nested axis; extra slots used to be
  ignored silently.
- Slice with `jax.lax.index_in_dim` rather than `jnp.take`, which lowers to a
  gather.

unroll_qwen_scanned_weights:
- Reject `layers_remainder`, as MaxTextToMaxTextConverter and raiden_unscan
  already do for a layer count that is not a multiple of the cycle, instead of
  leaving those layers unsynced.
- With an explicit `pattern_length`, put the global layer on slot
  `pattern_length - 1` and require `local_layers` to fill the slots before it.
- Use `jax.lax.index_in_dim` for both slices.
…yers

The scanned replay tests used a cycle of 2, so each block held a single local
layer and routing sent to the wrong local layer could not show. Add a cycle-3,
8-layer case -- two blocks of two local layers plus the global layer, and a
two-layer remainder -- that loads the scanned weights into the unscanned model
and requires the same loss for a distinct expert pair per layer. It also checks
that swapping two local layers' routing moves the unscanned loss, so the
comparison is able to catch a mix-up. Reversing the local slots inside the
block, or the remainder's two layers, fails it.
- moe: apply the expert-kernel optimization barrier only with scan_layers.
  Without a layer scan there is no loop to hoist the FSDP gather out of, so
  unscanned builds (vLLM's included) compile as they did before.
- Qwen3NextScannableBlock: drop the forced-routing slicing from the
  external-KV-cache path, which the decoder never reaches (it rejects forced
  routing with external caches), and raise if the block gets both. Also raise on
  per-layer (dict) attention_metadata: every scanned local layer has
  layer_idx=0, so a `layer.{i}` lookup would read the wrong layer's entry. vLLM,
  the only source of such metadata, runs unscanned.
- Merge the two contradictory comments on the load-balance loss.
- Document `apply_scanned_layers`' `xs` and drop the stale "only Gemma4 calls
  this" notes.
@NuojCheng
NuojCheng force-pushed the chengnuojin-qwen35-scan branch from 71a0b26 to f7ab0ff Compare October 21, 2026 17:49
@copybara-service
copybara-service Bot merged commit 9452e57 into main Oct 21, 2026
96 of 97 checks passed
@copybara-service
copybara-service Bot deleted the chengnuojin-qwen35-scan branch October 21, 2026 22:15
chiajunglien added a commit that referenced this pull request Oct 27, 2026
rl.yml replaces base.yml's logical_axis_rules instead of extending it, and it
was missing base.yml's `['local_layers', []]` rule. Since #5009, Qwen3.5 (like
Qwen3-Next and Gemma4) stacks the linear-attention layers of each scanned block
in a nested `local_layers` scan, so the actor's params carry that logical axis.

MaxText's own resolver leaves an axis without a rule unsharded, so the params are
placed as before. Tunix's PeftTrainer._shard_optimizer, however, derives the
shardings of the optimizer state and the gradient accumulator from
nnx.get_partition_spec under these rules, which keeps `local_layers` as is, and
tunix.sft.sharding_utils.get_sharding replicates any buffer whose PartitionSpec
names an axis that is not on the mesh. For Qwen3.5-35B-A3B that replicated
94 GiB of float32 gradients right before the first train step, and the
Pathways head (100 GiB limit) was OOMKilled.

Add the rule, and a CPU test that every axis of a scanned Qwen3.5 actor resolves
to mesh axes under rl.yml's rules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants