Repository navigation
Avoid MoE weight relayouts with FSDP prefetch via moe_flat_fsdp_weights - #5890
Merged
copybara-service[bot] merged 1 commit intoOct 23, 2026
Merged
Conversation
darisoy
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
huytransformer,
igorts-git,
jiangjy1982,
khatwanimohit,
michelle-yooh,
parambole,
richjames0,
shralex,
shuningjin,
vipannalla,
xibinliu and
zxhe-sean
as code owners
October 21, 2026 21:41
There was a problem hiding this comment.
Code Review
This pull request refactors FSDP prefetching and MoE layer configurations. It relaxes constraints for moe_flat_fsdp_weights, optimizes memory layout pinning during weight gathering, and updates fsdp_prefetch to use index-based slicing and dynamic updates inside jax.lax.scan instead of slicing and concatenating parameter trees. It also adds support for non-zero parameter scan axes. Feedback suggests using .get(a, 1) instead of direct dictionary lookup on ctx.mesh.shape to prevent potential KeyErrors.
darisoy
force-pushed
the
darisoy-fsdp-prefetch-avoid-relayouts
branch
from
October 21, 2026 21:47
d293f7b to
1fd0f4f
Compare
NuojCheng
reviewed
Oct 21, 2026
darisoy
force-pushed
the
darisoy-fsdp-prefetch-avoid-relayouts
branch
from
October 21, 2026 22:02
1fd0f4f to
091ce96
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
NuojCheng
approved these changes
Oct 21, 2026
darisoy
force-pushed
the
darisoy-fsdp-prefetch-avoid-relayouts
branch
from
October 22, 2026 01:08
091ce96 to
0633738
Compare
darisoy
force-pushed
the
darisoy-fsdp-prefetch-avoid-relayouts
branch
2 times, most recently
from
October 22, 2026 16:50
5572bf6 to
339dbd6
Compare
Closed
7 tasks done
darisoy
force-pushed
the
darisoy-fsdp-prefetch-avoid-relayouts
branch
from
October 22, 2026 17:09
339dbd6 to
5af48ea
Compare
Allows moe_flat_fsdp_weights=True without the explicit fp8 weight all-gather (bf16 and prefetch_fsdp_weights=True), so routed-expert weights are stored as 2D [E * rows, cols] and FSDP-gathered on their major-most dim. The gathered weight is then a free bitcast of the [E_local, rows, cols] gmm operand instead of a 3.5 GiB transpose per layer. fsdp_prefetch: pin the gathered-weight layout only for 3D+ weights gathered on their minor-most dim, and drop the per-layer sub-stack slicing / concatenation and the up-front moveaxis that added full-stack copies. Tests: unit test for _needs_minor_dim_layout_pin (now module level), and an abstract-model/sharding test for unquantized moe_flat_fsdp_weights with prefetch_fsdp_weights. fsdp_prefetch_inhomogeneous: its stage params are already layer-major, so the shared setup no longer moves the scan axis for them (scan_axis=0); flat routed-expert weights are covered by new Qwen3.5 parity tests. Qwen3.5 tests check that flat routed-expert weights are gathered per sub-layer over fsdp only (tagged, on dim 0, no layout pin) under expert parallelism.
darisoy
force-pushed
the
darisoy-fsdp-prefetch-avoid-relayouts
branch
from
October 22, 2026 17:10
5af48ea to
33a3932
Compare
7 tasks done
gobbleturk
approved these changes
Oct 23, 2026
igorts-git
approved these changes
Oct 23, 2026
copybara-service
Bot
deleted the
darisoy-fsdp-prefetch-avoid-relayouts
branch
October 23, 2026 20:00
darisoy
added a commit
that referenced
this pull request
Oct 26, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` and single-reduction SparseCore RS (#5973) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 27, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 28, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 29, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 30, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy
added a commit
that referenced
this pull request
Oct 31, 2026
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With
prefetch_fsdp_weights=True, the forward and backward passes were paying for full-size relayout copies of the routed-expert weights on every layer (b/576360738, roadmap PRE-05).Root cause. In the default 3D layout,
wois[E, mlp, embed]shardedP('expert', 'tensor', 'fsdp'), so FSDP shards its minor-most dim. On DeepSeek-V3 with 128-way FSDP the shard width is 56, smaller than the 128-wide TPU tile, so the all-gather / reduce-scatter run in layout{1, 0, 2}whilegmm/tgmmneed{2, 1, 0}. XLA inserts a 3.5 GiB transpose per layer (SparseCore 1sparse-core-data-format-callplus a TensorCorecopy). On top of that, the prefetch pipeline pinned every gathered weight's layout, which forced SparseCore relayouts onwi_0/wi_1as well, and built per-layer weight sub-stacks with slicing,jnp.concatenateand an up-frontjnp.moveaxis, which added full-stack TensorCore copies.Fix.
moe.py,types.py: allowmoe_flat_fsdp_weights=Truewithout the explicit fp8 weight all-gather. Theweight_gather=Falsebranch ofget_routed_moe_shardingsnow uses the flat(CompoundLogicalAxis(('exp', 'embed_moe')), ...)specs (FSDP then stripped as before), theNotImplementedErrorguard is removed, and the config check accepts unquantized configs andprefetch_fsdp_weights=Truein place of tokamaxgmm_v2. Routed-expert weights are stored as 2D[E * rows, cols]and gathered on their major-most dim, so thereshapeto[E_local, rows, cols]insideshard_mapis a free bitcast and the 3.5 GiB transposes disappear.fsdp_prefetch.py:_needs_minor_dim_layout_pin)._slice_aton a layer index) instead of passing sliced sub-stacks asxs, and write each layer's dW into a preallocated stack withdynamic_update_index_in_diminstead of concatenating parts.param_scan_axismoveaxisinto the custom-vjp setup (and its inverse onto the returned grads) so it is not materialized around the pipeline (_layer_major(..., move_scan_axis=False)for the homogeneous path).optimization_barrier, and drop the gathered weight from the per-step barriers.fsdp_prefetch_inhomogeneous.py(Prefetch FSDP weights per sub-layer for inhomogeneous blocks (Qwen3.5) #5865): its stage params are already layer-major before the shared_prefetch_setup, so it passesscan_axis=0to avoid moving the scan axis twice. Flat routed-expert weights go through the same per-sub-layer gathers (they are gathered on dim 0, so the minor-dim layout pin does not apply).Tests
deepseek3-671b, 15 layers, 128 experts,num_experts_per_tok=4, GhostFish 4x4x4 (64 chips),shard_mode=explicit,per_device_batch_size=2,max_target_length=4096,ici_fsdp_parallelism=-1,ici_expert_parallelism=1,sa_block_q/kv=1024,sa_block_dkv/dq=2048, tokamaxgmm_v2(use_tokamax_gmm=True,use_gmm_v2=True). Step time is the mean of 5 profiled steps; the other times are per step on a single core (/device:TPU:0).copyprefetch_fsdp_weights=False(xid/302727885)prefetch_fsdp_weights=True, 3D expert weights (xid/302729431)prefetch_fsdp_weights=True+moe_flat_fsdp_weights=True(xid/302730209){1, 0, 2}dwocopies). The remaining 3.5 GiB weight copies aregmm_v2's explicit backward RHS transposes (moe_gmm_v2_dlhs_transpose_rhs=False, the default), which the baseline has too.fsdp_prefetch_test.pyaddstest_needs_minor_dim_layout_pin(3D weight gathered on its minor-most dim is pinned; major-dim and 2D flat weights are not)moe_test.pyaddsFlatFsdpWeightsGraphTest.test_unquantized_prefetch_abstract_model_and_shardings(bf16,prefetch_fsdp_weights=True, no tokamaxgmm_v2: flat[E * rows, cols]shapes and('expert', 'fsdp')dim-0 sharding). The existing flat-weight fp8 parity test and thefsdp_prefetch_test.pygradient-parity tests still apply.nnx_decoders_test.pyaddstest_qwen3_5_moe_flat_fsdp_weights(and aparam_scan_axis=0variant): Qwen3.5 per-sub-layer prefetching with flat routed-expert weights matches the plain scanned stack in loss and gradients.test_qwen3_5_flat_routed_expert_gathers_under_expert_parallelismchecks, on an fsdp=4 x expert=2 mesh, that each of the 12 flat routed-expert weights of the 4 sub-layers gets onexla_explicit_fsdpall-gather on dim 0 overfsdponly, stays sharded overexpert, and gets no layout pin.qwen3.5-35b-a3b(inhomogeneous per-sub-layer prefetch from #5865), GhostFish 4x4x4 (64 chips),shard_mode=explicit,per_device_batch_size=1,max_target_length=1024,ici_fsdp_parallelism=-1,remat_policy=custom, tokamaxgmm_v2(use_tokamax_gmm=True,use_gmm_v2=True). Same measurement method as above.prefetch_fsdp_weights=False(xid/303041931)prefetch_fsdp_weights=True, 3D expert weights (xid/303038743)prefetch_fsdp_weights=True+moe_flat_fsdp_weights=True(xid/303080050)fsdp_prefetch_inhomogeneous.pystill have same-layout loop-carry copies of these weights; removing those is left for a follow-up.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.