Skip to content

Avoid MoE weight relayouts with FSDP prefetch via moe_flat_fsdp_weights - #5890

Merged
copybara-service[bot] merged 1 commit into
mainfrom
darisoy-fsdp-prefetch-avoid-relayouts
Oct 23, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
darisoy-fsdp-prefetch-avoid-relayouts

Conversation

@darisoy

@darisoy darisoy commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Description

With prefetch_fsdp_weights=True, the forward and backward passes were paying for full-size relayout copies of the routed-expert weights on every layer (b/576360738, roadmap PRE-05).

Root cause. In the default 3D layout, wo is [E, mlp, embed] sharded P('expert', 'tensor', 'fsdp'), so FSDP shards its minor-most dim. On DeepSeek-V3 with 128-way FSDP the shard width is 56, smaller than the 128-wide TPU tile, so the all-gather / reduce-scatter run in layout {1, 0, 2} while gmm / tgmm need {2, 1, 0}. XLA inserts a 3.5 GiB transpose per layer (SparseCore 1 sparse-core-data-format-call plus a TensorCore copy). On top of that, the prefetch pipeline pinned every gathered weight's layout, which forced SparseCore relayouts on wi_0 / wi_1 as well, and built per-layer weight sub-stacks with slicing, jnp.concatenate and an up-front jnp.moveaxis, which added full-stack TensorCore copies.

Fix.

  • moe.py, types.py: allow moe_flat_fsdp_weights=True without the explicit fp8 weight all-gather. The weight_gather=False branch of get_routed_moe_shardings now uses the flat (CompoundLogicalAxis(('exp', 'embed_moe')), ...) specs (FSDP then stripped as before), the NotImplementedError guard is removed, and the config check accepts unquantized configs and prefetch_fsdp_weights=True in place of tokamax gmm_v2. Routed-expert weights are stored as 2D [E * rows, cols] and gathered on their major-most dim, so the reshape to [E_local, rows, cols] inside shard_map is a free bitcast and the 3.5 GiB transposes disappear.
  • fsdp_prefetch.py:
    • Pin the gathered-weight layout only for 3D+ weights whose FSDP gather is on their minor-most dim (module-level _needs_minor_dim_layout_pin).
    • Index layer weights directly in the scans (_slice_at on a layer index) instead of passing sliced sub-stacks as xs, and write each layer's dW into a preallocated stack with dynamic_update_index_in_dim instead of concatenating parts.
    • Move the param_scan_axis moveaxis into the custom-vjp setup (and its inverse onto the returned grads) so it is not materialized around the pipeline (_layer_major(..., move_scan_axis=False) for the homogeneous path).
    • Put only the two peeled layer slices (instead of the whole parameter stack) in the backward entry optimization_barrier, and drop the gathered weight from the per-step barriers.
  • fsdp_prefetch_inhomogeneous.py (Prefetch FSDP weights per sub-layer for inhomogeneous blocks (Qwen3.5) #5865): its stage params are already layer-major before the shared _prefetch_setup, so it passes scan_axis=0 to avoid moving the scan axis twice. Flat routed-expert weights go through the same per-sub-layer gathers (they are gathered on dim 0, so the minor-dim layout pin does not apply).

Tests

deepseek3-671b, 15 layers, 128 experts, num_experts_per_tok=4, GhostFish 4x4x4 (64 chips), shard_mode=explicit, per_device_batch_size=2, max_target_length=4096, ici_fsdp_parallelism=-1, ici_expert_parallelism=1, sa_block_q/kv=1024, sa_block_dkv/dq=2048, tokamax gmm_v2 (use_tokamax_gmm=True, use_gmm_v2=True). Step time is the mean of 5 profiled steps; the other times are per step on a single core (/device:TPU:0).

Run Step time Peak HBM Exposed SC data-format wait TC copy Exposed FSDP AG wait Exposed FSDP RS wait XProf
Baseline, prefetch_fsdp_weights=False (xid/302727885) 2702.9 ms 38.2 GiB 12.3 ms 276.9 ms 380.2 ms 393.0 ms trace
prefetch_fsdp_weights=True, 3D expert weights (xid/302729431) 2489.4 ms 58.5 GiB 116.5 ms 313.1 ms 18.8 ms 345.5 ms trace
prefetch_fsdp_weights=True + moe_flat_fsdp_weights=True (xid/302730209) 2172.4 ms 53.2 GiB 57.0 ms 150.3 ms 19.9 ms 268.2 ms trace
  • Step time: -530.5 ms (-19.6%) vs. the baseline and -317.0 ms (-12.7%) vs. prefetch with 3D expert weights.
  • With flat weights, no FSDP relayout copies of the MoE weights or weight gradients remain (no copies around the all-gathers / reduce-scatters, no layout-constraint copies, no {1, 0, 2} dwo copies). The remaining 3.5 GiB weight copies are gmm_v2's explicit backward RHS transposes (moe_gmm_v2_dlhs_transpose_rhs=False, the default), which the baseline has too.
  • On the megablox (v1) kernel the same change also helps: 2648.3 ms (xid/299683407, prefetch before this PR) -> 2298.2 ms (xid/302656685).
  • Unit tests: fsdp_prefetch_test.py adds test_needs_minor_dim_layout_pin (3D weight gathered on its minor-most dim is pinned; major-dim and 2D flat weights are not) moe_test.py adds FlatFsdpWeightsGraphTest.test_unquantized_prefetch_abstract_model_and_shardings (bf16, prefetch_fsdp_weights=True, no tokamax gmm_v2: flat [E * rows, cols] shapes and ('expert', 'fsdp') dim-0 sharding). The existing flat-weight fp8 parity test and the fsdp_prefetch_test.py gradient-parity tests still apply.
  • nnx_decoders_test.py adds test_qwen3_5_moe_flat_fsdp_weights (and a param_scan_axis=0 variant): Qwen3.5 per-sub-layer prefetching with flat routed-expert weights matches the plain scanned stack in loss and gradients. test_qwen3_5_flat_routed_expert_gathers_under_expert_parallelism checks, on an fsdp=4 x expert=2 mesh, that each of the 12 flat routed-expert weights of the 4 sub-layers gets one xla_explicit_fsdp all-gather on dim 0 over fsdp only, stays sharded over expert, and gets no layout pin.
  • Qwen3.5-35B-A3B AoT on GF 4x4x4 with prefetch: flat weights remove all 8 FSDP relayout copies of the 512 MiB routed-expert weights (around the per-sub-layer all-gathers) that 3D weights have; compiled temp HBM 14.09 -> 13.51 GiB.

qwen3.5-35b-a3b (inhomogeneous per-sub-layer prefetch from #5865), GhostFish 4x4x4 (64 chips), shard_mode=explicit, per_device_batch_size=1, max_target_length=1024, ici_fsdp_parallelism=-1, remat_policy=custom, tokamax gmm_v2 (use_tokamax_gmm=True, use_gmm_v2=True). Same measurement method as above.

Run Step time Peak HBM Exposed SC data-format wait TC data formatting SC1 data formatting XProf
Baseline, prefetch_fsdp_weights=False (xid/303041931) 1303.2 ms 7.75 GiB 40.1 ms 146.9 ms 602 ms trace
prefetch_fsdp_weights=True, 3D expert weights (xid/303038743) 1164.6 ms 12.39 GiB 30.2 ms 155.5 ms 63 ms trace
prefetch_fsdp_weights=True + moe_flat_fsdp_weights=True (xid/303080050) 1085.1 ms 12.88 GiB 15.6 ms 136.2 ms 31 ms trace
  • Step time: -218.1 ms (-16.7%) vs. the baseline and -79.5 ms (-6.8%) vs. prefetch with 3D expert weights.
  • Flat weights halve the exposed SparseCore data-format wait and the SparseCore data-formatting time, since the 512 MiB routed-expert weights no longer need relayouts around their all-gathers.
  • Peak HBM is 0.5 GiB (about one 512 MiB routed-expert weight) higher than with 3D weights. The per-sub-layer scans in fsdp_prefetch_inhomogeneous.py still have same-layout loop-carry copies of these weights; removing those is left for a follow-up.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have added or updated tests accordingly.
  • I have updated documentation accordingly.
  • I have included unit tests for new features or bug fixes.
  • I have added unit test and integration test coverage for all modified code paths.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors FSDP prefetching and MoE layer configurations. It relaxes constraints for moe_flat_fsdp_weights, optimizes memory layout pinning during weight gathering, and updates fsdp_prefetch to use index-based slicing and dynamic updates inside jax.lax.scan instead of slicing and concatenating parameter trees. It also adds support for non-zero parameter scan axes. Feedback suggests using .get(a, 1) instead of direct dictionary lookup on ctx.mesh.shape to prevent potential KeyErrors.

Comment thread src/maxtext/layers/fsdp_prefetch.py Resolved
@darisoy
darisoy force-pushed the darisoy-fsdp-prefetch-avoid-relayouts branch from d293f7b to 1fd0f4f Compare October 21, 2026 21:47
Comment thread src/maxtext/layers/moe.py Resolved
@darisoy
darisoy force-pushed the darisoy-fsdp-prefetch-avoid-relayouts branch from 1fd0f4f to 091ce96 Compare October 21, 2026 22:02
@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.22642% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/fsdp_prefetch.py 95.65% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@darisoy
darisoy force-pushed the darisoy-fsdp-prefetch-avoid-relayouts branch from 091ce96 to 0633738 Compare October 22, 2026 01:08
@darisoy
darisoy force-pushed the darisoy-fsdp-prefetch-avoid-relayouts branch 2 times, most recently from 5572bf6 to 339dbd6 Compare October 22, 2026 16:50
@darisoy
darisoy force-pushed the darisoy-fsdp-prefetch-avoid-relayouts branch from 339dbd6 to 5af48ea Compare October 22, 2026 17:09
Allows moe_flat_fsdp_weights=True without the explicit fp8 weight all-gather (bf16 and prefetch_fsdp_weights=True), so routed-expert weights are stored as 2D [E * rows, cols] and FSDP-gathered on their major-most dim. The gathered weight is then a free bitcast of the [E_local, rows, cols] gmm operand instead of a 3.5 GiB transpose per layer.

fsdp_prefetch: pin the gathered-weight layout only for 3D+ weights gathered on their minor-most dim, and drop the per-layer sub-stack slicing / concatenation and the up-front moveaxis that added full-stack copies.

Tests: unit test for _needs_minor_dim_layout_pin (now module level), and an abstract-model/sharding test for unquantized moe_flat_fsdp_weights with prefetch_fsdp_weights.

fsdp_prefetch_inhomogeneous: its stage params are already layer-major, so the shared setup no longer moves the scan axis for them (scan_axis=0); flat routed-expert weights are covered by new Qwen3.5 parity tests.
Qwen3.5 tests check that flat routed-expert weights are gathered per sub-layer over fsdp only (tagged, on dim 0, no layout pin) under expert parallelism.
@darisoy
darisoy force-pushed the darisoy-fsdp-prefetch-avoid-relayouts branch from 5af48ea to 33a3932 Compare October 22, 2026 17:10
@copybara-service
copybara-service Bot merged commit 0585207 into main Oct 23, 2026
80 checks passed
@copybara-service
copybara-service Bot deleted the darisoy-fsdp-prefetch-avoid-relayouts branch October 23, 2026 20:00
darisoy added a commit that referenced this pull request Oct 26, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` and single-reduction SparseCore RS (#5973)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 27, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 28, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 29, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 30, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 31, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants