Repository navigation
Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan - #5009
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements hierarchical nested scans for Qwen3-Next and Qwen3.5 hybrid decoders, enabling per-layer rematerialization to optimize memory usage. It introduces scannable blocks that structure inhomogeneous attention cycles and updates checkpoint conversion utilities to support multi-axis stacked tensors with arbitrary nesting depths. Comprehensive unit tests are added to verify decoder parity and state restoration. The review feedback suggests updating the type annotations for mt_key from str to str | tuple[str, ...] in stacked_axes and _build_multi_axis_stacked_tensor to accurately reflect that composite keys can be represented as tuples.
| return depth | ||
|
|
||
|
|
||
| def stacked_axes(mt_key: str, config, depth: int) -> tuple: |
There was a problem hiding this comment.
The mt_key parameter is annotated as str, but the function explicitly handles tuple inputs (lines 69-70). To ensure type safety and prevent static analysis warnings, the type annotation should be updated to str | tuple[str, ...].
| def stacked_axes(mt_key: str, config, depth: int) -> tuple: | |
| def stacked_axes(mt_key: str | tuple[str, ...], config, depth: int) -> tuple: |
| tensor_getter_fn: Callable[[str], np.ndarray], | ||
| hook_fns: Any, | ||
| target_leaf: Any, | ||
| config, | ||
| mt_key: str = "", |
| @@ -344,66 +344,59 @@ def get_maxtext_model_info(config): | |||
|
|
|||
|
|
|||
| def _build_multi_axis_stacked_tensor( | |||
| hf_source_keys: List[List[str]], | |||
| hf_source_keys: List[Any], | |||
| tensor_getter_fn: Callable[[str], np.ndarray], | |||
| hook_fns: Any, | |||
| target_shape: tuple, | |||
| config, | |||
| mt_key: str = "", | |||
4c37a30 to
ec280c7
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
ec280c7 to
8f902ef
Compare
682b3d7 to
2892a98
Compare
2892a98 to
ee342ff
Compare
2a7f4c1 to
7d1536b
Compare
|
|
||
|
|
||
| class Qwen3_5ScannableBlock(nnx.Module): | ||
| class Qwen3_5ScannableBlock(Qwen3NextScannableBlock): |
There was a problem hiding this comment.
Scanned Qwen3.5 checkpoints change layout: layers/layer_{slot} becomes layers/local_layers + layers/global_layer. Old scanned checkpoints won't load or convert. Can you note this in the PR description with a migration path, or keep a fallback for the old keys?
There was a problem hiding this comment.
Agree, scan-format changes gonna impact previous checkpoints. The ideal way is to re-run the to_maxtext.py to reproduce checkpoints.
| prefix = f"params-decoder-layers-layer_{block_idx}" | ||
| # The length-1 _scan_global_layer scan is a memory boundary, not a param stack, so | ||
| # global_layer params stay flat [block] while local_layers nest [block][local]. | ||
| num_blocks = num_main_layers // layer_cycle_interval |
There was a problem hiding this comment.
When the layer count isn't a multiple of the cycle, the model now builds layers_remainder, but nothing maps it. Those layers fail on MaxText→HF and are skipped on HF→MaxText. Can you add a mapping for layers_remainder-local_layers-*, or raise a clear error? Same at L1565 for Qwen3-Next.
| @@ -2684,19 +2718,22 @@ def _apply_qwen3_next_scanned_blocks(self, y, layer_args, layer_kwargs, kv_cache | |||
| length=scan_length, | |||
| kv_caches_stacked=grouped_kv_caches, | |||
| skip_block_remat=True, | |||
There was a problem hiding this comment.
This passes skip_block_remat=True, but with prefetch_fsdp_weights=True, _apply_layers_sequentially hands off to the prefetch path (L1207) without it. Qwen3.5 blocks remat their own sub-layers, so they get rematerialized twice. Can you forward skip_block_remat there?
| def test_per_layer_routing_reaches_the_scan_remainder_block(self): | ||
| """Layers past the last whole cycle live in a separate remainder block.""" | ||
| seq_len, batch_size, top_k, num_experts = 8, 1, 2, 4 | ||
| cycle_interval, layers = 2, 3 |
There was a problem hiding this comment.
With cycle = 2 there's only one local layer per block, so these tests can't catch routing that's sent to the wrong sub-layer. Can you add a case with cycle >= 3 and a remainder, checking that scanned and unscanned losses match? Covering Qwen3-Next too would be nice.
| model_mode=self.model_mode, | ||
| quant=self.quant, | ||
| lambda layer_rngs: self._make_decoder_layer( | ||
| layer_idx=0, |
There was a problem hiding this comment.
nit: Every local layer gets layer_idx=0, so dict-based attention_metadata maps them all to "layer.0". Probably fine since vLLM runs unscanned, but worth a check.
| if self.global_layer is not None: | ||
| global_kv = kv_cache[self.num_local] if (kv_cache is not None and self.num_local < len(kv_cache)) else None | ||
| y, new_kv = self._run_layer(self.global_layer, y, layer_kwargs, global_kv) | ||
| global_routing = None if forced_routed_experts is None else forced_routed_experts[self.num_local] |
There was a problem hiding this comment.
nit: This looks unreachable: the decoder raises when forced routing and external KV caches are combined. Add a comment or remove it?
| ) | ||
| mlp_output, load_balance_loss = self._apply_mlp(hidden_states, deterministic, forced_routed_experts) | ||
|
|
||
| # We sow the load balancing loss so it can be collected and added to the total loss |
There was a problem hiding this comment.
nit: "We sow the load balancing loss" and "Assigned, not sown" contradict each other. Merge into one comment?
| param_scan_axis: int, | ||
| apply_fn: Callable[[nnx.Module, Any], Any], | ||
| apply_fn: Callable[..., Any], | ||
| xs: Any | None = None, |
There was a problem hiding this comment.
nit: Please document the new xs argument in the docstring. "Gemma4 is currently the only caller" is also out of date.
| mt_key = mt_key[0] | ||
| if isinstance(mt_key, str) and "-local_layers" in mt_key: | ||
| param_scan_axis = config.param_scan_axis | ||
| nested_axes = (param_scan_axis, param_scan_axis + 1) |
There was a problem hiding this comment.
nit: With param_scan_axis=0 and FP8 experts, this returns (0, 0, 1), and moveaxis fails on the repeated axis. Assert param_scan_axis >= 1, or handle 0.
There was a problem hiding this comment.
It now raises ValueError
darisoy
left a comment
There was a problem hiding this comment.
LGTM overall.
biggest concern is around breaking the existing qwen3.5 checkpoints
Qwen3.5's decoder layer and scannable block were near-duplicates of
Qwen3-Next's. Collapse them onto the Qwen3-Next implementations, which
already nest a scan over the cycle's local layers inside a trip-count-one
scan around the full-attention layer, so Qwen3.5 picks up per-sub-layer
remat instead of block-level remat.
Sharing is via three overridable factory hooks on Qwen3NextDecoderLayer
(`_make_full_attention`, `_make_linear_attention`, `_make_mlp`). Qwen3.5
overrides only those, keeping the submodule attribute names -- and
therefore the checkpoint parameter paths -- identical, so the whole
forward pass, both norms and the residual structure are inherited rather
than copied. qwen3_5.py drops from 266 to 127 lines.
Two things had to be repaired to keep functionality identical:
Router replay. `_apply_layers_sequentially` filters call kwargs against
the callee signature, and Qwen3NextScannableBlock did not accept
`forced_routed_experts`, so scanned Qwen3.5 lost it. Thread it through
`nnx_scan.apply_scanned_layers` (new optional per-layer `xs`), the block
(scan xs for the local layers, closure for the global one, per-layer
slices on the external-KV unroll) and the hybrid scan entry point, which
also slices the scan remainder. Combining forced routing with
externally-managed KV caches now raises rather than silently dropping
one of them.
vLLM weight sync. The RL rollout converter and the direct-sync unroll
both assumed one module per cycle slot (`layers.layer_{slot}`) with a
single stacked axis. The nested block stores `local_layers` with the
cycle slot on `param_scan_axis + 1` and `global_layer` with the block
axis alone, so both now read that layout, selecting the slot once per
plan group. Trainer states in the old layout still convert. This also
fixes Qwen3-Next, which produced the nested layout already and was
therefore never syncable.
AOT compile, per_device_batch_size=1, max_target_length=1024, flash
attention with tokamax splash, temp_size_in_bytes vs main:
qwen3.5-35b-a3b v5p-128 16.16 GB -> 11.03 GB (-31.7%)
qwen3.5-35b-a3b v5p-512 11.99 GB -> 6.74 GB (-43.8%)
qwen3.5-397b-a17b v5p-512 75.62 GB -> 46.01 GB (-39.2%)
The routed-MoE kernel pspecs drop the fsdp axis so XLA emits the FSDP all-gather at the sharding constraint in sparse_matmul. Under a scanned decoder the kernels reaching that constraint are a per-layer dynamic-slice of a stacked param, and with nothing in between XLA rewrites all_gather(dynamic_slice(w)) into dynamic_slice(all_gather(w)). That gather is loop invariant, so it is hoisted out of the scan and every layer's gathered expert kernel is live at once -- num_layers * num_experts * embed * moe_mlp * dtype bytes of extra peak memory. The jnp.asarray(kernel, dtype) cast in __call__ normally separates the slice from the collective and blocks the rewrite. It is an identity when weight_dtype == dtype, which is why the blow-up only shows up there: qwen3-next-80b-a3b at tpu7x-128 compiles to 14.20 GB of temporaries with weight_dtype=float32 but 65.59 GB with weight_dtype=bfloat16, on code that is byte-identical to main's. Stand an optimization_barrier in for the cast in exactly that case, matching the barrier moe_fsdp_use_two_stage_all_gather already uses a few lines above. AOT temp_size_in_bytes per device: model topology before after qwen3-next-80b-a3b tpu7x-128 65.59 GB 12.11 GB qwen3.5-35b-a3b tpu7x-128 30.23 GB 6.94 GB qwen3.5-35b-a3b tpu7x-16 35.71 GB 17.04 GB qwen3-30b-a3b tpu7x-16 13.03 GB 12.42 GB Gating on the identity keeps every weight_dtype=float32 build byte-identical (qwen3.5-35b-a3b 25132249440 at tpu7x-16 and 8510193056 at tpu7x-128, qwen3-next-80b-a3b 14204451200 at tpu7x-128, before and after). Applying the barrier unconditionally costs 3-8% on those.
pylint C3001 unnecessary-lambda-assignment on the no-router-replay branch of Qwen3NextScannableBlock._scan_local_layers. It was the only finding failing the Code Quality Check / Pre-commit Linters job. Mirrors the def already used in the sibling branch; behaviour is unchanged.
Cut the explanatory comments and docstring prose added by this branch down to the few that record something the code cannot show. 180 lines removed, 27 added; no code changes (AST identical, docstrings excluded).
`_scan_global_layer` carried the global-attention layer's params through the scan carry and returned them unchanged, which makes them loop invariants of the trip-count-one while loop. Recent XLA hoists the layer's FSDP expert weight all-gathers out of that loop into the enclosing block scan, where they run exposed before the local-layer scan and keep the gathered weights live across it. Passing the params as scan xs (with a leading length-1 axis) keeps them indexed by the loop, so each all-gather stays inside the global layer's loop. Verified in TPU AoT (Qwen3.5-35B-A3B, gf=4x4x4, FSDP over all chips): with the carry, the forward block body holds the three routed-expert all-gathers; with xs, it holds none and the global layer's loop holds all of its all-gathers. Compiled temp memory is unchanged.
`_scan_global_layer` feeds the global layer's params in as scan xs and drops every `nnx.Param` on the way out, so a param materialized while tracing the layer (e.g. a Qwix LoRA adapter or quantization param) never reached `nnx.update` and was silently lost. Before params moved to xs they were in the carry, so this would at least have failed with a structure mismatch. Mirror `nnx_scan.apply_scanned_layers`: drop only the params that were fed in, return the rest as an extra scan output, and apply them with `nnx.update`. When no params are created that extra output is an empty pytree. Adds a regression test (run for Qwen3-Next and Qwen3.5) that creates a param inside the global layer's call and checks it lands on the block; it fails without this change.
When num_decoder_layers is not a multiple of inhomogeneous_layer_cycle_interval, the scanned decoder keeps the layers past the last whole cycle in a `layers_remainder` block whose `local_layers` are stacked on param_scan_axis alone. Nothing mapped those params, so MaxText -> HF failed on them and HF -> MaxText skipped them: a 6-layer, cycle-4 Qwen3.5 or Qwen3-Next mapped 36 of its 53 params. Map `layers_remainder-local_layers-*` (and register its hooks) for both models, and stop `stacked_axes` from treating remainder keys as a nested block scan, so remainder routed experts keep the plain (experts, layers) layout. Both models now map 53/53 params at 6 layers, and still map exactly with no remainder keys at 8. `stacked_axes` also raises a clear error for expert-stacked weights in a nested block scan with param_scan_axis=0, where the expert axis no longer leads and the returned axes used to repeat, failing inside np.moveaxis. The type annotations of `mt_key` now admit composite (tuple) keys.
`_apply_layers_sequentially` handed the prefetch path everything but `skip_block_remat`. The prefetch pipeline rematerializes each scanned module, so for a block that already rematerializes its own sub-layers (Qwen3-Next, Qwen3.5, Gemma4, DeepSeek-V4) the backward pass recomputed the whole block and then, in the block's own remat, every sub-layer again. Simply dropping the pipeline's remat for such blocks is not an option: the blocks' inner scans read their weights as scan xs (`moveaxis(w)` for `local_layers`, `w[None]` for the global layer), so `jax.vjp` of the bare block keeps them as residuals -- 35 residuals worth 101% of the block's weight bytes for a Qwen3.5 block, i.e. every block's gathered weights saved between the passes. Instead, forward `skip_block_remat` and have the pipeline extend the remat policy to also save each sub-layer's `decoder_layer_input`, which Qwen3NextDecoderLayer now names like the other decoder layers do. The forward then keeps the sub-layer inputs (2 residuals per Qwen3.5 block, no weights) and the backward recomputes every sub-layer once. On the 8-layer CPU test config with remat_policy=full, compiled value_and_grad FLOPs of the prefetch path drop from 5.36e8 to 4.30e8: 4 forward-equivalents per block instead of 5, the same as the regular scan. Loss and gradients match the non-prefetch path exactly for Qwen3.5, Qwen3-Next and Gemma4.
MaxTextToMaxTextConverter: - Take a `local_layers` source's cycle slot before casting it. Every slot's plan group reads the whole source, so casting first cast the full stack once per slot (3x the work at cycle 4). - Require exactly `cycle - 1` slots on the nested axis; extra slots used to be ignored silently. - Slice with `jax.lax.index_in_dim` rather than `jnp.take`, which lowers to a gather. unroll_qwen_scanned_weights: - Reject `layers_remainder`, as MaxTextToMaxTextConverter and raiden_unscan already do for a layer count that is not a multiple of the cycle, instead of leaving those layers unsynced. - With an explicit `pattern_length`, put the global layer on slot `pattern_length - 1` and require `local_layers` to fill the slots before it. - Use `jax.lax.index_in_dim` for both slices.
…yers The scanned replay tests used a cycle of 2, so each block held a single local layer and routing sent to the wrong local layer could not show. Add a cycle-3, 8-layer case -- two blocks of two local layers plus the global layer, and a two-layer remainder -- that loads the scanned weights into the unscanned model and requires the same loss for a distinct expert pair per layer. It also checks that swapping two local layers' routing moves the unscanned loss, so the comparison is able to catch a mix-up. Reversing the local slots inside the block, or the remainder's two layers, fails it.
- moe: apply the expert-kernel optimization barrier only with scan_layers.
Without a layer scan there is no loop to hoist the FSDP gather out of, so
unscanned builds (vLLM's included) compile as they did before.
- Qwen3NextScannableBlock: drop the forced-routing slicing from the
external-KV-cache path, which the decoder never reaches (it rejects forced
routing with external caches), and raise if the block gets both. Also raise on
per-layer (dict) attention_metadata: every scanned local layer has
layer_idx=0, so a `layer.{i}` lookup would read the wrong layer's entry. vLLM,
the only source of such metadata, runs unscanned.
- Merge the two contradictory comments on the load-balance loss.
- Document `apply_scanned_layers`' `xs` and drop the stale "only Gemma4 calls
this" notes.
71a0b26 to
f7ab0ff
Compare
rl.yml replaces base.yml's logical_axis_rules instead of extending it, and it was missing base.yml's `['local_layers', []]` rule. Since #5009, Qwen3.5 (like Qwen3-Next and Gemma4) stacks the linear-attention layers of each scanned block in a nested `local_layers` scan, so the actor's params carry that logical axis. MaxText's own resolver leaves an axis without a rule unsharded, so the params are placed as before. Tunix's PeftTrainer._shard_optimizer, however, derives the shardings of the optimizer state and the gradient accumulator from nnx.get_partition_spec under these rules, which keeps `local_layers` as is, and tunix.sft.sharding_utils.get_sharding replicates any buffer whose PartitionSpec names an axis that is not on the mesh. For Qwen3.5-35B-A3B that replicated 94 GiB of float32 gradients right before the first train step, and the Pathways head (100 GiB limit) was OOMKilled. Add the rule, and a CPU test that every axis of a scanned Qwen3.5 actor resolves to mesh axes under rl.yml's rules.
What
Extends the Qwen3-Next scanned-block work from #4964 to Qwen3.5.
Qwen3.5 repeats the exact same hybrid attention period as Qwen3-Next —
inhomogeneous_layer_cycle_interval = 4, i.e. three GatedDeltaNet layers followed by one full-attention layer — so rather than porting the logic, this makes Qwen3.5 reuse it.The two architectures now share the decoder layer as well as the block.
Qwen3_5DecoderLayersubclassesQwen3NextDecoderLayerand overrides only three factory hooks —_make_full_attention,_make_linear_attention,_make_mlp— so the forward pass, both LayerNorms, the residual structure, theis_full_attention_layerderivation and the MoE load-balance handling are inherited rather than duplicated. The hooks keep the submodule attribute names (input_layernorm,attention,post_attention_layernorm,mlp), so the checkpoint parameter paths are identical either way.Qwen3_5ScannableBlocklikewise subclassesQwen3NextScannableBlockand overrides only_make_decoder_layer.models/qwen3_5.pygoes from 266 to 127 lines. Qwen3.5 picks up the inner scan over the homogeneous linear-attention layers, the trip-count-onelax.scanscheduling barrier around the full-attention layer, and per-sub-layer remat, for free.Changes
models/qwen3.py— extract sub-block construction fromQwen3NextDecoderLayer.__init__into_make_full_attention()/_make_linear_attention()/_make_mlp(), and layer construction fromQwen3NextScannableBlock.__init__into_make_decoder_layer(). These are the only points a sibling architecture has to diverge at. Also threadsforced_routed_expertsthrough the layer and the block (see below). No behavior change for Qwen3-Next.models/qwen3_5.py—Qwen3_5DecoderLayerandQwen3_5ScannableBlockbecome thin subclasses that override the hooks.Qwen3NextDecoderLayergains an explicitis_full_attention_layerkwarg, because inside a scan a sub-layer's position in the cycle is not recoverable fromlayer_idx; it still falls back to the(layer_idx + 1) % cycle == 0derivation when unscanned. The MoE load-balance loss is assigned rather thansow-n —sowappends to a tuple, so the layer'sIntermediatestructure grows on every call, which alax.scanbody cannot express.layers/decoders.py,layers/nnx_decoders.py— routeQWEN3_5through the shared hybrid path._init_scanned_qwen3_next/_apply_qwen3_next_scanned_blocksare renamed to..._qwen3_hybrid_...and pick the block class bydecoder_block, including forlayers_remainder.forced_routed_experts_per_layer()is factored out ofreshape_forced_routed_experts_for_scan()so the hybrid path can slice routing per sub-layer.layers/nnx_scan.py—apply_scanned_layers()gains an optionalxspytree of per-layer scan inputs. Callers that pass noxskeep the two-argumentapply_fnsignature; Gemma4, the only other caller, is unchanged.layers/moe.py— hold the FSDP expert-kernel all-gather inside the decoder loop whenweight_dtype == dtype. Not part of the port; it is a pre-existing bug in the scanned MoE path that this PR would otherwise widen to Qwen3.5. See Theweight_dtype=bfloat16blow-up below.checkpoint_conversion/utils/param_mapping.py— remap Qwen3.5's scanned checkpoint mapping from the old per-cycle-positionlayers-layer_{i}prefixes to thelocal_layers/global_layertree, mirroring what perf(qwen3-next): nested scan over hybrid attention with per-layer remat #4964 did for Qwen3-Next. Local layers nest[block][local], the global layer is flat[block].checkpoint_conversion/utils/tensor_handling.py—stacked_axes()was guarded onisinstance(mt_key, str), so Qwen3.5's composite(wi_0, wi_1)key (fed by one fused HFgate_up_projtensor) fell through to the leading-axis MoE layout instead of the nested-scan layout and would have silently transposed those weights. Tuple keys now resolve via their first component.integration/vllm/weight_converter.py,integration/vllm/maxtext_vllm_rollout.py— teach the RL rollout weight sync to read the nested tree (see below).Results
AOT
temp_size_in_bytesper device, using the config fromtrain_compile_test.py::test_qwen3_5(per_device_batch_size=1,max_target_length=1024,sparse_matmul,megablox,attention=flash,use_tokamax_splash,remat_policy=full— thebase.ymldefault — on both sides). Both sides freshly compiled againstmain@457040f7:Exact bytes: 35b/tpu7x-16
30359507040 → 25132249440; 35b/tpu7x-843612869024 → 44113347456; 397b/v5p-51275622737696 → 46011566848. The 397b/v5p-128 OOM row is the original pre-rebase measurement and was not re-run.The saving is a roughly constant 5–7 GB, not a constant percentage. Sweeping device count on identical silicon (qwen3.5-35b-a3b, tpu7x-N):
What shrinks with device count is the baseline, not the saving. The FSDP-sharded share of the working set falls as 1/N, while what this PR removes does not shard at all:
sparse_matmuldeliberately stripsfsdpfrom the expert-kernel pspecs, so the gathered per-device MoE kernels are full size on every chip, and block-level remat keeps all four sub-layers' worth of them co-live. Per-sub-layer remat keeps one. The same absolute saving is therefore 17% of a 30 GB budget and 45% of a 16 GB one.The cross-generation control agrees: v5p-128 and tpu7x-64 are both 64 JAX devices (v5p slice names count TensorCores) and start from quite different baselines — 16.16 GB and 19.81 GB — yet save 5.129 GB and 5.105 GB, within 0.5% of each other. v5p-512 (256 devices) saves 5.25 GB, 11.99 → 6.74 GB.
argument_size_in_bytesis unchanged in both directions (26.01 GB at 35b/tpu7x-16, 18.61 GB at 397b/v5p-512). Generated code size roughly halves on every row, including the ones where temp memory does not move — that is the nested scan replacing an unrolled cycle, and it is topology-independent: 35b/tpu7x-8 205 → 99 MB, 35b/tpu7x-16 210 → 99 MB, 397b/v5p-512 320 → 163 MB.Real training on a v7-8 (qwen3.5-35b-a3b, synthetic data,
ici_fsdp_parallelism=8,per_device_batch_size=1,max_target_length=1024, 6 steps) converges on both sides with matching loss curves (step 5: 7.056 before, 7.072 after — RNG-fork differences only). Memory there is a wash, 40.6 → 41.1 GB estimated temp, consistent with the tpu7x-8 row.The layer-sharing refactor is free. Recompiling qwen3.5-35b-a3b at tpu7x-16 after collapsing
Qwen3_5DecoderLayerontoQwen3NextDecoderLayerand threading router replay givestemp_size_in_bytes=25132249440,generated_code_size_in_bytes=99265536— byte-identical to the build before the refactor.Test plan
All CPU-only, run individually against this branch:
tests/unit/nnx_decoders_test.py-k Qwen3: 17 passed, up from 7 onmain)tests/unit/moe_test.pytests/unit/router_replay_test.pytests/unit/nnx_scan_test.pytests/unit/param_mapping_test.pytests/post_training/unit/weight_converter_test.pynnx_decoders_test.pyare parameterized by class attributes so the Qwen3.5 subclasses reuse them: nested-scan output matches a sequential unroll numerically, and the Linen and NNX parameter trees match, both for a whole number of periods and with a remainder. New: a block given forced routing that does not cover every sub-layer must raise.moe_test.pyaddstest_sparse_matmul_pins_expert_kernel_all_gather_when_the_upcast_is_an_identity, parameterized over(bf16, bf16),(f32, f32)and(f32, bf16). It asserts the barrier is applied exactly when the upcast is an identity, and — by object identity, since the stand-in kernels share a shape — that the sharding constraint consumes the barrier's outputs rather than the raw kernels, so a barrier whose results are dropped does not pass. Both matching-dtype cases fail without themoe.pychange.param_mapping_test.pycovers thelocal_layers/global_layermapping shape and the composite-key axis fix.router_replay_test.pyadds a case for the scan remainder block, where only the trailing layer's routing differs between two runs, so the losses can diverge only if the remainder replays its own slice. Reverting theqwen3.py/nnx_scan.py/nnx_decoders.pychanges failstest_loss_fn_with_forced_routed_experts_scanned_qwen3_5andtest_per_layer_routing_is_not_broadcast_scanned.weight_converter_test.pyadds a nested-layout source tree and asserts it converts byte-identically to the per-slot layout, plus a truncated cycle-slot axis being rejected. Both fail if theweight_converter.pychange is reverted.tests/post_training/unit/vllm_rollout_unroll_test.pyadds a nestedlocal_layers+global_layerunroll case and a missing-slot-axis rejection. This file needsvllminstalled to import the module under test, so it was verified in CI-equivalent form by executing the unroll helpers in isolation: the nested layout produces a denselayers_{0..7}with every slot on its own layer, and the legacy per-slot and homogeneous paths are unchanged. The same check fails againstmain.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.