Skip to content

Prefetch FSDP weights per sub-layer for inhomogeneous blocks (Qwen3.5) - #5865

Merged
copybara-service[bot] merged 1 commit into
mainfrom
darisoy-qwen35-inhomogeneous-prefetch
Oct 22, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
darisoy-qwen35-inhomogeneous-prefetch

Conversation

@darisoy

@darisoy darisoy commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

With prefetch_fsdp_weights=True, fsdp_prefetch prefetches whole scan steps. For Qwen3.5 a scan step is a block of 3 Gated DeltaNet sub-layers plus 1 global-attention sub-layer, so whole-step prefetching gathers an entire block's weights at once. This PR adds fsdp_prefetch_inhomogeneous, which prefetches such blocks one sub-layer at a time while keeping the block's nested scan structure from #5009.

  • Forward: the all-gather of sub-layer j+1 overlaps the compute of sub-layer j, across block boundaries (the last sub-layer of a block prefetches sub-layer 0 of the next block).

  • Backward: the all-gather of sub-layer j-1 overlaps the recompute + vjp of sub-layer j, followed by a reduce-scatter of sub-layer j's weight gradients.

  • Memory: at most two sub-layers of gathered weights are live in the forward pass and three in the backward pass, as in fsdp_prefetch.

  • Nested scans are preserved: consecutive sub-layers with the same structure form a group, and every sub-layer runs in a while loop nested in the block scan (trip-count-one loops use skip-simplify-while-loops_trip-count-one). Within a group of n sub-layers, the n-1 that prefetch a sub-layer of the same group share one scan; the remaining one prefetches a sub-layer of another structure in its own trip-count-one loop. For Qwen3.5:

    • forward: [GDN 0, GDN 1] -> [GDN 2] -> [attention]
    • backward: [attention] -> [GDN 2, GDN 1] -> [GDN 0]

    The loop boundaries keep each prefetched all-gather inside the sub-layer that issues it, so XLA cannot complete it ahead of the previous sub-layer's compute.

  • Gathers, reduce-scatters and remat-policy handling are shared with fsdp_prefetch (small refactor there to expose them).

  • NNXDecoder._apply_layers_sequentially_prefetch takes this path only when every block param lives in the block's sub-layers (local_layers + global_layer, or a block defining prefetch_sublayer_names()); homogeneous decoders and other blocks keep using fsdp_prefetch unchanged. skip_block_remat (from Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan #5009) only applies to that whole-step path, since this path already rematerializes each sub-layer on its own.

  • Expert parallelism: the gathers come from fsdp_prefetch, so the expert/context co-sharded axes from Gather expert/context co-sharded axes in FSDP weight prefetching (EP support) #5848 are gathered here too (one tagged all-gather over ('fsdp', 'expert') for non-routed weights; routed experts keep expert and gather only fsdp).

Files

  • src/maxtext/layers/fsdp_prefetch_inhomogeneous.py (new): sub-layer grouping and the nested-scan prefetch pipeline (jax.custom_vjp).
  • src/maxtext/layers/fsdp_prefetch.py: factor _prepare_prefetch_call and _layer_major out of apply_layers_with_fsdp_prefetch so the new module can reuse them (together with the existing _prefetch_setup). No behavior change for homogeneous stacks.
  • src/maxtext/layers/nnx_decoders.py: route inhomogeneous blocks to the new pipeline.
  • tests/unit/nnx_decoders_test.py: loss/grad parity tests against the plain scanned stack (1–4 blocks; 1, 2, 3 and 5 Gated DeltaNet sub-layers per block; remat_policy=minimal; param_scan_axis=0), a forward-only (no grad) parity test, routing/fallback tests (homogeneous decoders and global-only blocks keep whole-step prefetching), direct tests of the sub-layer splitting, and two expert-parallelism tests: loss/grad parity on an fsdp=2 x expert=2 mesh (skipped below 4 devices) and a device-free check that lowers each sub-layer weight's gather on an abstract explicit fsdp=4 x expert=2 mesh. Update Qwen3.5 scan-remat pattern by re-using Qwen3-Next's nested block scan #5009's test_blocks_with_internal_remat_save_their_sub_layer_inputs now forces whole-step prefetching, since Qwen3.5 blocks take the per-sub-layer path.

Tests

HLO (TPU AoT, Qwen3.5-35B-A3B, gf=4x4x4, FSDP over all chips): the forward block loop contains three nested sub-layer loops ([GDN 0–1] with 2 trips, [GDN 2], [attention]) and the backward block loop mirrors them. The block loop bodies contain no FSDP all-gathers; each sub-layer loop issues and completes its own prefetch gathers.

Expert parallelism (TPU AoT, Qwen3.5-397B-A17B with 8 layers, v5p-512, FSDP 32 x expert 8, shard_mode=explicit): compiles with prefetch_fsdp_weights=True; 83 of the 91 all-gathers in the train step carry the xla_explicit_fsdp tag, i.e. every decoder weight gather (the other 8 are the embedding table and small integer gathers). With prefetching off there are no tagged gathers.

Step time (GF 4x4x4, steady state, steps 4/5/9; the baseline includes the #5009 _scan_global_layer fix):

Recipe Baseline (prefetch_fsdp_weights=False) This PR (prefetch_fsdp_weights=True)
per_device_batch_size=1, max_target_length=1024 1.439 s (xid/301894631, trace) 1.360 s (xid/301873347, trace)

With this recipe the per-sub-layer compute is short relative to the weight all-gathers, and prefetching makes the step about 5.5% faster.

Common flags: remat_policy=custom shard_mode=explicit scan_layers=True opt_type=sgd weight_dtype=bfloat16 ici_fsdp_parallelism=-1; the prefetch run adds --xla_tpu_enable_fsdp_latency_hiding_scheduler=true --xla_tpu_explicit_fsdp_dedicated_sparse_core_id=0.

Unit tests: XLA_FLAGS=--xla_force_host_platform_device_count=4 JAX_PLATFORMS=cpu pytest tests/unit/nnx_decoders_test.py -k "Prefetch or prefetch or remat or global_layer": 61 passed (includes the existing whole-step prefetch tests).

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have added or updated tests accordingly.
  • I have updated documentation accordingly.
  • I have included unit tests for new features or bug fixes.
  • I have added unit test and integration test coverage for all modified code paths.

@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.12664% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/fsdp_prefetch_inhomogeneous.py 99.04% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces fsdp_prefetch_inhomogeneous.py to support FSDP weight prefetching for inhomogeneous scanned blocks, such as Qwen3.5, by prefetching sub-layers individually rather than as whole scan steps. The implementation includes a new nested VJP pipeline for these blocks and updates nnx_decoders.py and the unit tests to integrate and verify this functionality. Several review comments were addressed regarding the deprecation of jax.tree.map in favor of jax.tree_util.tree_map and the use of jnp.unstack for better compatibility.

Comment thread src/maxtext/layers/fsdp_prefetch.py Resolved
@darisoy
darisoy force-pushed the darisoy-qwen35-inhomogeneous-prefetch branch 3 times, most recently from 96d2c78 to 14fbcc3 Compare October 21, 2026 17:35
@NuojCheng
NuojCheng force-pushed the chengnuojin-qwen35-scan branch from 71a0b26 to f7ab0ff Compare October 21, 2026 17:49
@darisoy
darisoy force-pushed the darisoy-qwen35-inhomogeneous-prefetch branch 3 times, most recently from a6d16ba to 20fc607 Compare October 21, 2026 21:10
Base automatically changed from chengnuojin-qwen35-scan to main October 21, 2026 22:15
Adds `fsdp_prefetch_inhomogeneous`: with `prefetch_fsdp_weights=True`, a scanned
block made of sub-layers with different parameter structures (Qwen3.5's
`local_layers` Gated DeltaNet scan plus one global-attention `global_layer`, or
any block that defines `prefetch_sublayer_names()`) is prefetched one sub-layer
at a time instead of one whole block at a time.

Forward: the all-gather of sub-layer j+1 overlaps the compute of sub-layer j,
across block boundaries. Backward: the all-gather of sub-layer j-1 overlaps the
recompute + vjp of sub-layer j, followed by a reduce-scatter of sub-layer j's
weight gradients. The forward pass holds at most two sub-layers of gathered
weights and the backward pass at most three.

The block keeps its nested scan structure: consecutive sub-layers with the same
structure form a group, and every sub-layer runs in a while loop nested in the
block scan (trip-count-one loops keep `skip-simplify-while-loops_trip-count-one`).
Within a group of n sub-layers, the n-1 that prefetch a sub-layer of the same
group share one scan; the remaining one prefetches a sub-layer of another
structure in its own trip-count-one loop. For Qwen3.5:

  forward:  [GDN 0, GDN 1] -> [GDN 2] -> [attention]
  backward: [attention] -> [GDN 2, GDN 1] -> [GDN 0]

The loop boundaries keep each prefetched all-gather inside the sub-layer that
issues it. The gathers, reduce-scatters and remat policy handling are shared
with `fsdp_prefetch`.

`NNXDecoder` only takes this path when every block param lives in the listed
sub-layers; homogeneous decoders and other blocks keep prefetching whole scan
steps with `fsdp_prefetch`.
@darisoy
darisoy force-pushed the darisoy-qwen35-inhomogeneous-prefetch branch from 20fc607 to 45b0b34 Compare October 21, 2026 22:22
@copybara-service
copybara-service Bot merged commit 66bc270 into main Oct 22, 2026
74 of 76 checks passed
@copybara-service
copybara-service Bot deleted the darisoy-qwen35-inhomogeneous-prefetch branch October 22, 2026 00:50
darisoy added a commit that referenced this pull request Oct 26, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` and single-reduction SparseCore RS (#5973)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 27, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 28, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 29, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 30, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 31, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants