Repository navigation
Prefetch FSDP weights per sub-layer for inhomogeneous blocks (Qwen3.5) - #5865
Merged
copybara-service[bot] merged 1 commit intoOct 22, 2026
Merged
Conversation
darisoy
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
huytransformer,
igorts-git,
jiangjy1982,
khatwanimohit,
parambole,
richjames0,
shralex,
shuningjin,
vipannalla and
xibinliu
as code owners
October 20, 2026 18:31
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Code Review
This pull request introduces fsdp_prefetch_inhomogeneous.py to support FSDP weight prefetching for inhomogeneous scanned blocks, such as Qwen3.5, by prefetching sub-layers individually rather than as whole scan steps. The implementation includes a new nested VJP pipeline for these blocks and updates nnx_decoders.py and the unit tests to integrate and verify this functionality. Several review comments were addressed regarding the deprecation of jax.tree.map in favor of jax.tree_util.tree_map and the use of jnp.unstack for better compatibility.
darisoy
force-pushed
the
darisoy-qwen35-inhomogeneous-prefetch
branch
3 times, most recently
from
October 21, 2026 17:35
96d2c78 to
14fbcc3
Compare
NuojCheng
force-pushed
the
chengnuojin-qwen35-scan
branch
from
October 21, 2026 17:49
71a0b26 to
f7ab0ff
Compare
NuojCheng
requested review from
jacoguzo,
michelle-yooh and
zxhe-sean
as code owners
October 21, 2026 17:49
darisoy
force-pushed
the
darisoy-qwen35-inhomogeneous-prefetch
branch
3 times, most recently
from
October 21, 2026 21:10
a6d16ba to
20fc607
Compare
Adds `fsdp_prefetch_inhomogeneous`: with `prefetch_fsdp_weights=True`, a scanned block made of sub-layers with different parameter structures (Qwen3.5's `local_layers` Gated DeltaNet scan plus one global-attention `global_layer`, or any block that defines `prefetch_sublayer_names()`) is prefetched one sub-layer at a time instead of one whole block at a time. Forward: the all-gather of sub-layer j+1 overlaps the compute of sub-layer j, across block boundaries. Backward: the all-gather of sub-layer j-1 overlaps the recompute + vjp of sub-layer j, followed by a reduce-scatter of sub-layer j's weight gradients. The forward pass holds at most two sub-layers of gathered weights and the backward pass at most three. The block keeps its nested scan structure: consecutive sub-layers with the same structure form a group, and every sub-layer runs in a while loop nested in the block scan (trip-count-one loops keep `skip-simplify-while-loops_trip-count-one`). Within a group of n sub-layers, the n-1 that prefetch a sub-layer of the same group share one scan; the remaining one prefetches a sub-layer of another structure in its own trip-count-one loop. For Qwen3.5: forward: [GDN 0, GDN 1] -> [GDN 2] -> [attention] backward: [attention] -> [GDN 2, GDN 1] -> [GDN 0] The loop boundaries keep each prefetched all-gather inside the sub-layer that issues it. The gathers, reduce-scatters and remat policy handling are shared with `fsdp_prefetch`. `NNXDecoder` only takes this path when every block param lives in the listed sub-layers; homogeneous decoders and other blocks keep prefetching whole scan steps with `fsdp_prefetch`.
darisoy
force-pushed
the
darisoy-qwen35-inhomogeneous-prefetch
branch
from
October 21, 2026 22:22
20fc607 to
45b0b34
Compare
NuojCheng
approved these changes
Oct 21, 2026
gobbleturk
approved these changes
Oct 21, 2026
copybara-service
Bot
deleted the
darisoy-qwen35-inhomogeneous-prefetch
branch
October 22, 2026 00:50
This was referenced Oct 22, 2026
darisoy
added a commit
that referenced
this pull request
Oct 26, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` and single-reduction SparseCore RS (#5973) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 27, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 28, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 29, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 30, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 31, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With
prefetch_fsdp_weights=True,fsdp_prefetchprefetches whole scan steps. For Qwen3.5 a scan step is a block of 3 Gated DeltaNet sub-layers plus 1 global-attention sub-layer, so whole-step prefetching gathers an entire block's weights at once. This PR addsfsdp_prefetch_inhomogeneous, which prefetches such blocks one sub-layer at a time while keeping the block's nested scan structure from #5009.Forward: the all-gather of sub-layer j+1 overlaps the compute of sub-layer j, across block boundaries (the last sub-layer of a block prefetches sub-layer 0 of the next block).
Backward: the all-gather of sub-layer j-1 overlaps the recompute + vjp of sub-layer j, followed by a reduce-scatter of sub-layer j's weight gradients.
Memory: at most two sub-layers of gathered weights are live in the forward pass and three in the backward pass, as in
fsdp_prefetch.Nested scans are preserved: consecutive sub-layers with the same structure form a group, and every sub-layer runs in a while loop nested in the block scan (trip-count-one loops use
skip-simplify-while-loops_trip-count-one). Within a group of n sub-layers, the n-1 that prefetch a sub-layer of the same group share one scan; the remaining one prefetches a sub-layer of another structure in its own trip-count-one loop. For Qwen3.5:[GDN 0, GDN 1] -> [GDN 2] -> [attention][attention] -> [GDN 2, GDN 1] -> [GDN 0]The loop boundaries keep each prefetched all-gather inside the sub-layer that issues it, so XLA cannot complete it ahead of the previous sub-layer's compute.
Gathers, reduce-scatters and remat-policy handling are shared with
fsdp_prefetch(small refactor there to expose them).NNXDecoder._apply_layers_sequentially_prefetchtakes this path only when every block param lives in the block's sub-layers (local_layers+global_layer, or a block definingprefetch_sublayer_names()); homogeneous decoders and other blocks keep usingfsdp_prefetchunchanged.skip_block_remat(from Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan #5009) only applies to that whole-step path, since this path already rematerializes each sub-layer on its own.Expert parallelism: the gathers come from
fsdp_prefetch, so theexpert/contextco-sharded axes from Gather expert/context co-sharded axes in FSDP weight prefetching (EP support) #5848 are gathered here too (one tagged all-gather over('fsdp', 'expert')for non-routed weights; routed experts keepexpertand gather onlyfsdp).Files
src/maxtext/layers/fsdp_prefetch_inhomogeneous.py(new): sub-layer grouping and the nested-scan prefetch pipeline (jax.custom_vjp).src/maxtext/layers/fsdp_prefetch.py: factor_prepare_prefetch_calland_layer_majorout ofapply_layers_with_fsdp_prefetchso the new module can reuse them (together with the existing_prefetch_setup). No behavior change for homogeneous stacks.src/maxtext/layers/nnx_decoders.py: route inhomogeneous blocks to the new pipeline.tests/unit/nnx_decoders_test.py: loss/grad parity tests against the plain scanned stack (1–4 blocks; 1, 2, 3 and 5 Gated DeltaNet sub-layers per block;remat_policy=minimal;param_scan_axis=0), a forward-only (no grad) parity test, routing/fallback tests (homogeneous decoders and global-only blocks keep whole-step prefetching), direct tests of the sub-layer splitting, and two expert-parallelism tests: loss/grad parity on anfsdp=2 x expert=2mesh (skipped below 4 devices) and a device-free check that lowers each sub-layer weight's gather on an abstract explicitfsdp=4 x expert=2mesh. Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan #5009'stest_blocks_with_internal_remat_save_their_sub_layer_inputsnow forces whole-step prefetching, since Qwen3.5 blocks take the per-sub-layer path.Tests
HLO (TPU AoT, Qwen3.5-35B-A3B,
gf=4x4x4, FSDP over all chips): the forward block loop contains three nested sub-layer loops ([GDN 0–1]with 2 trips,[GDN 2],[attention]) and the backward block loop mirrors them. The block loop bodies contain no FSDP all-gathers; each sub-layer loop issues and completes its own prefetch gathers.Expert parallelism (TPU AoT, Qwen3.5-397B-A17B with 8 layers,
v5p-512, FSDP 32 x expert 8,shard_mode=explicit): compiles withprefetch_fsdp_weights=True; 83 of the 91 all-gathers in the train step carry thexla_explicit_fsdptag, i.e. every decoder weight gather (the other 8 are the embedding table and small integer gathers). With prefetching off there are no tagged gathers.Step time (GF 4x4x4, steady state, steps 4/5/9; the baseline includes the #5009
_scan_global_layerfix):prefetch_fsdp_weights=False)prefetch_fsdp_weights=True)per_device_batch_size=1,max_target_length=1024With this recipe the per-sub-layer compute is short relative to the weight all-gathers, and prefetching makes the step about 5.5% faster.
Common flags:
remat_policy=custom shard_mode=explicit scan_layers=True opt_type=sgd weight_dtype=bfloat16 ici_fsdp_parallelism=-1; the prefetch run adds--xla_tpu_enable_fsdp_latency_hiding_scheduler=true --xla_tpu_explicit_fsdp_dedicated_sparse_core_id=0.Unit tests:
XLA_FLAGS=--xla_force_host_platform_device_count=4 JAX_PLATFORMS=cpu pytest tests/unit/nnx_decoders_test.py -k "Prefetch or prefetch or remat or global_layer": 61 passed (includes the existing whole-step prefetch tests).Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.