Repository navigation
Use unroll=2 sub-layer scans in inhomogeneous FSDP weight prefetching - #5922
Conversation
The same-group sub-layer scans of fsdp_prefetch_inhomogeneous carried one gathered-weight buffer, so each step's all-gather of the next sub-layer landed in a fresh buffer that XLA copied into the loop carry (a full-size TensorCore copy of every gathered weight, e.g. three 512 MiB routed-expert weights per Qwen3.5-35B-A3B GDN sub-layer step, in both passes). Same-group scans with >= 2 steps now run two sub-layers per loop trip and alternate between two gathered-weight buffers, with the optimization-barrier ordering of fsdp_prefetch, so the all-gather reuses the loop-carry buffer. _sublayer_scan unrolls by hand: jax.lax.scan(unroll=2) drops the while-loop entirely for a single 2-step trip, which moves the copy to the block loop carry instead. Tests: Qwen3.5 parity with 4 local layers per block (2-step trip plus a 1-step remainder), and a check that same-group scans unroll by 2.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request optimizes the FSDP prefetch mechanism for inhomogeneous blocks by introducing a double-buffering strategy. By unrolling same-group scans by two, the system can alternate between gathered-weight buffers, preventing unnecessary XLA copies into the loop-carry buffer. This change significantly improves step time and reduces memory overhead for models like Qwen3.5. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` and single-reduction SparseCore RS (#5973) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
…on/fsdp_prefetch.md`) - Add forward-pass timeline diagrams without and with weight prefetching - Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922) - Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994) - Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags - Update DeepSeek-V3 and Qwen3.5 benchmark tables
Description
Stacked on #5890. Part of b/574100720 (roadmap PRE-03, avoid buffer copies in FSDP prefetch).
Brings the
unroll=2scheme of the homogeneous prefetch pipeline (#5709) to the per-sub-layer prefetch of inhomogeneous blocks insrc/maxtext/layers/fsdp_prefetch_inhomogeneous.py(e.g. Qwen3.5). As in #5709, the same-group sub-layer scans of both passes now run two sub-layers per loop iteration and alternate between two gathered-weight buffers, so each step's prefetched weight lands in the loop-carry buffer instead of being copied into it.unroll=2same-group scans in forward and backward:n >= 3sub-layers (n - 1 >= 2same-group steps, e.g. Qwen3.5's 3 Gated DeltaNet sub-layers) runs its same-group scan withunroll=2; stepjoverlapsgather(W(j+1))with sub-layerj, ordered viajax.lax.optimization_barrier((y, params_next))soW(j-1)is dead beforeAG(j+1)starts andAG(j+1)reuses its buffer.unroll=2; stepjoverlapsgather(W(j-1))with recompute + VJP of sub-layerj, with the sameoptimization_barrier((g, y, saved, params_prev))before the gather andoptimization_barrier((g, dw))after the reduce-scatter as infsdp_prefetch._custom_vjp_prefetch_pipeline._sublayer_scan(..., unroll=2)unrolls by hand: each loop trip runs two consecutive steps, and an odd remaining step runs in its own single-trip loop. Unlikefsdp_prefetch, a same-group scan here often has a single 2-step trip (Qwen3.5: 2 same-group steps per block), andjax.lax.scan(..., unroll=2)emits no while-loop at all in that case: the steps would be inlined into the block loop and the copies would only move to the block loop carry. The hand-unrolled scan keeps the sub-layer while-loop (withskip-simplify-while-loops_trip-count-one) like the other sub-layer loops.Before this PR, each same-group step returned the all-gather of the next sub-layer, issued while the current weight was still being read, so XLA copied its result into the loop carry at the end of every step. On Qwen3.5-35B-A3B with
moe_flat_fsdp_weights=Truethese were three 512 MiB routed-expert weight copies (wi_0,wi_1,wo) per GDN sub-layer step in each pass, about 44 ms per step of exposed TensorCore time plus one extra live 512 MiB buffer.Tests
qwen3.5-35b-a3b, GhostFish 4x4x4 (64 chips),shard_mode=explicit,per_device_batch_size=1,max_target_length=1024,ici_fsdp_parallelism=-1,remat_policy=custom, tokamaxgmm_v2(use_tokamax_gmm=True,use_gmm_v2=True),prefetch_fsdp_weights=True. Step time is the mean of 5 profiled steps; the other times are per step on a single core (/device:TPU:0).prefetch_fsdp_weights=False(xid/303041931)moe_flat_fsdp_weights=True(xid/303080050)moe_flat_fsdp_weights=True(xid/303149411)bf16[131072,2048]/bf16[524288,512], 3 in the forward and 3 in the backward same-group loop) are gone, and peak HBM drops by 422 MiB. AoT on the same config: compiled temp HBM 13.29 -> 12.85 GB.gmm_v2's explicit backward RHS transposes (moe_gmm_v2_dlhs_transpose_rhs=False, the default), which the baseline has too.nnx_decoders_test.pyaddstest_qwen3_5_four_local_layers_per_block(3 same-group steps: one 2-step trip plus a 1-step remainder) andtest_qwen3_5_same_group_scans_unroll_by_two(the 2-step same-group scans of both passes unroll by 2; single-step loops do not). AllTestNNXDecoderPrefetchInhomogeneousloss and gradient parity tests pass.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.