Skip to content

Use unroll=2 sub-layer scans in inhomogeneous FSDP weight prefetching - #5922

Merged
copybara-service[bot] merged 1 commit into
mainfrom
darisoy-fsdp-prefetch-inhomog-double-buffer
Oct 23, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
darisoy-fsdp-prefetch-inhomog-double-buffer

Conversation

@darisoy

@darisoy darisoy commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Stacked on #5890. Part of b/574100720 (roadmap PRE-03, avoid buffer copies in FSDP prefetch).

Brings the unroll=2 scheme of the homogeneous prefetch pipeline (#5709) to the per-sub-layer prefetch of inhomogeneous blocks in src/maxtext/layers/fsdp_prefetch_inhomogeneous.py (e.g. Qwen3.5). As in #5709, the same-group sub-layer scans of both passes now run two sub-layers per loop iteration and alternate between two gathered-weight buffers, so each step's prefetched weight lands in the loop-carry buffer instead of being copied into it.

  • unroll=2 same-group scans in forward and backward:
    • Forward: a group with n >= 3 sub-layers (n - 1 >= 2 same-group steps, e.g. Qwen3.5's 3 Gated DeltaNet sub-layers) runs its same-group scan with unroll=2; step j overlaps gather(W(j+1)) with sub-layer j, ordered via jax.lax.optimization_barrier((y, params_next)) so W(j-1) is dead before AG(j+1) starts and AG(j+1) reuses its buffer.
    • Backward: the reverse same-group scan runs with unroll=2; step j overlaps gather(W(j-1)) with recompute + VJP of sub-layer j, with the same optimization_barrier((g, y, saved, params_prev)) before the gather and optimization_barrier((g, dw)) after the reduce-scatter as in fsdp_prefetch._custom_vjp_prefetch_pipeline.
    • Single-step loops (the sub-layer that prefetches another group, and groups with fewer than 3 sub-layers) are unchanged.
  • _sublayer_scan(..., unroll=2) unrolls by hand: each loop trip runs two consecutive steps, and an odd remaining step runs in its own single-trip loop. Unlike fsdp_prefetch, a same-group scan here often has a single 2-step trip (Qwen3.5: 2 same-group steps per block), and jax.lax.scan(..., unroll=2) emits no while-loop at all in that case: the steps would be inlined into the block loop and the copies would only move to the block loop carry. The hand-unrolled scan keeps the sub-layer while-loop (with skip-simplify-while-loops_trip-count-one) like the other sub-layer loops.

Before this PR, each same-group step returned the all-gather of the next sub-layer, issued while the current weight was still being read, so XLA copied its result into the loop carry at the end of every step. On Qwen3.5-35B-A3B with moe_flat_fsdp_weights=True these were three 512 MiB routed-expert weight copies (wi_0, wi_1, wo) per GDN sub-layer step in each pass, about 44 ms per step of exposed TensorCore time plus one extra live 512 MiB buffer.

Tests

qwen3.5-35b-a3b, GhostFish 4x4x4 (64 chips), shard_mode=explicit, per_device_batch_size=1, max_target_length=1024, ici_fsdp_parallelism=-1, remat_policy=custom, tokamax gmm_v2 (use_tokamax_gmm=True, use_gmm_v2=True), prefetch_fsdp_weights=True. Step time is the mean of 5 profiled steps; the other times are per step on a single core (/device:TPU:0).

Run Step time Peak HBM TC data formatting Exposed SC data-format wait XProf
Baseline, prefetch_fsdp_weights=False (xid/303041931) 1303.2 ms 7.75 GiB 146.9 ms 40.1 ms trace
#5890, moe_flat_fsdp_weights=True (xid/303080050) 1085.1 ms 12.88 GiB 136.2 ms 15.6 ms trace
This PR, moe_flat_fsdp_weights=True (xid/303149411) 1038.7 ms 12.47 GiB 84.0 ms 14.4 ms trace
  • Step time: -46.4 ms (-4.3%) vs. Avoid MoE weight relayouts with FSDP prefetch via moe_flat_fsdp_weights #5890 and -264.5 ms (-20.3%) vs. the baseline.
  • The 6 loop-carry copies of the 512 MiB flat routed-expert weights (bf16[131072,2048] / bf16[524288,512], 3 in the forward and 3 in the backward same-group loop) are gone, and peak HBM drops by 422 MiB. AoT on the same config: compiled temp HBM 13.29 -> 12.85 GB.
  • Of the remaining 84 ms of TensorCore data formatting, about 61 ms is gmm_v2's explicit backward RHS transposes (moe_gmm_v2_dlhs_transpose_rhs=False, the default), which the baseline has too.
  • Unit tests: nnx_decoders_test.py adds test_qwen3_5_four_local_layers_per_block (3 same-group steps: one 2-step trip plus a 1-step remainder) and test_qwen3_5_same_group_scans_unroll_by_two (the 2-step same-group scans of both passes unroll by 2; single-step loops do not). All TestNNXDecoderPrefetchInhomogeneous loss and gradient parity tests pass.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have added or updated tests accordingly.
  • I have updated documentation accordingly.
  • I have included unit tests for new features or bug fixes.
  • I have added unit test and integration test coverage for all modified code paths.

The same-group sub-layer scans of fsdp_prefetch_inhomogeneous carried one gathered-weight buffer, so each step's all-gather of the next sub-layer landed in a fresh buffer that XLA copied into the loop carry (a full-size TensorCore copy of every gathered weight, e.g. three 512 MiB routed-expert weights per Qwen3.5-35B-A3B GDN sub-layer step, in both passes).

Same-group scans with >= 2 steps now run two sub-layers per loop trip and alternate between two gathered-weight buffers, with the optimization-barrier ordering of fsdp_prefetch, so the all-gather reuses the loop-carry buffer. _sublayer_scan unrolls by hand: jax.lax.scan(unroll=2) drops the while-loop entirely for a single 2-step trip, which moves the copy to the block loop carry instead.

Tests: Qwen3.5 parity with 4 local layers per block (2-step trip plus a 1-step remainder), and a check that same-group scans unroll by 2.
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request optimizes the FSDP prefetch mechanism for inhomogeneous blocks by introducing a double-buffering strategy. By unrolling same-group scans by two, the system can alternate between gathered-weight buffers, preventing unnecessary XLA copies into the loop-carry buffer. This change significantly improves step time and reduces memory overhead for models like Qwen3.5.

Highlights

  • Double-buffering for FSDP prefetch: Implemented double-buffering for per-sub-layer FSDP prefetch in inhomogeneous blocks (e.g., Qwen3.5) to eliminate redundant weight copies and reduce peak HBM usage.
  • Unrolled scan implementation: Added an unroll=2 option to _sublayer_scan, allowing same-group scans to alternate between two gathered-weight buffers and reuse memory effectively.
  • Performance improvements: Achieved a 4.3% reduction in step time compared to the previous implementation and a 20.3% reduction against the baseline, while reducing peak HBM by 422 MiB.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@darisoy darisoy changed the title Double-buffer gathered weights in Qwen3.5 per-sub-layer FSDP prefetch Use unroll=2 sub-layer scans in inhomogeneous FSDP weight prefetching Oct 23, 2026
@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.12195% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/fsdp_prefetch_inhomogeneous.py 95.12% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Base automatically changed from darisoy-fsdp-prefetch-avoid-relayouts to main October 23, 2026 20:00
@copybara-service
copybara-service Bot merged commit 1ef9934 into main Oct 23, 2026
79 checks passed
@copybara-service
copybara-service Bot deleted the darisoy-fsdp-prefetch-inhomog-double-buffer branch October 23, 2026 20:37
darisoy added a commit that referenced this pull request Oct 26, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` and single-reduction SparseCore RS (#5973)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 27, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 28, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 29, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 30, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
darisoy added a commit that referenced this pull request Oct 31, 2026
…on/fsdp_prefetch.md`)

- Add forward-pass timeline diagrams without and with weight prefetching
- Document inhomogeneous nested sub-layer scans and `unroll=2` double buffering (#5865, #5922)
- Document `prefetch_fsdp_defer_grad_reduce_scatter=True` for homogeneous and inhomogeneous blocks (#5973, #5994)
- Document expert/context co-sharding (#5848), `moe_flat_fsdp_weights=True` (#5890), `moe_gmm_v2_dlhs_transpose_rhs=True`, `moe_use_direct_token_gather=True`, and XLA flags
- Update DeepSeek-V3 and Qwen3.5 benchmark tables
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants