Skip to content

Add explicit FSDP weight prefetching for m3 ScannedLayers - #6405

Open
AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-remat-hybrid-barriersfrom
yaoyuchen/m3-fsdp-weight-prefetch
Open

AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-remat-hybrid-barriersfrom
yaoyuchen/m3-fsdp-weight-prefetch

Conversation

@AusarYao28

Copy link
Copy Markdown
Collaborator

Description

Depends on #6249 (stacked on yaoyuchen/m3-remat-hybrid-barriers; review the top commit only).

Inside scanned m3 decoder layers (ScannedLayers), XLA latency-hiding flags alone produce byte-identical HLOs on TPU7x and leave per-layer FSDP weight all-gathers serialized or split across windowed einsums. This PR adds explicit double-buffered FSDP weight prefetching (and optional deferred gradient reduce-scatter) for m3 ScannedLayers, off by default and byte-identical in compiled HLO when disabled.

  • Adds src/maxtext/m3/infra/fsdp_prefetch.py (FsdpPrefetchRunner, apply_fsdp_prefetch, maybe_apply_fsdp_prefetch) implementing a path-transparent jax.custom_vjp unroll=2 double-buffered scan pipeline supporting auto and explicit meshes, multi-axis sharding (2D FSDP, FSDP+TP, FSDP+DP), SparseCore 1 pinning on multi-trip TPU7x scan loops (num_layers >= 6, avoiding the TPU7x ComputeEmitter all-gather(%convert) crash on inlined <6-layer loops), host-stack residual streaming for minimal_offloaded remat policies (failing closed on residual leaf-count mismatch), fail-closed rejection of remat_policy='none', and safe no-op fallback for num_layers <= 1, non-FSDP meshes, decode mode, and parameter_memory_host_offload=True.
  • Adds an optional fsdp_prefetch hook on ScannedLayers in src/maxtext/m3/infra/scan.py (order-invariant with scan and zero new imports in scan.py). Qwen3.8 cycle scans are intentionally untouched here and will follow after #6325.
  • Adds prefetch_fsdp_weights and prefetch_fsdp_defer_grad_reduce_scatter (both default False) to TrainerConfig in src/maxtext/m3/configs/config.py, wired into both src/maxtext/utils/model_creation_utils.py and src/maxtext/m3/train/loop.py.
  • Known limitation (documented on the config flag): combining minimal_offloaded with prefetch_fsdp_weights=True compiles and matches numerics, but synchronous host-stack dynamic_update_slice stalls PCIe DMA on TPU7x (815 vs 244 ms/step on tpu7x-8).

Tests

  • CPU unit tests (4-device host mesh, serial):
    • tests/unit/m3/fsdp_prefetch_test.py: 24 passed (covers forward/backward parity across explicit, auto, multi-axis, stateful, and minimal_offloaded configs, fail-closed checks, dual-driver wiring, compiled-HLO byte-identity when disabled, and overlap HLO structure when enabled).
    • Regression suites: scan_tests.py (7 passed), train_loop_test.py (10 passed), m3_smoke_test.py (3 passed), model_sharding_test.py (76 passed, 3 skipped), remat_test.py (29 passed).
    • Negative control: slicing layer 0 instead of layer 1 in the prefetch carry fails explicit_full_pf_drs (loss 4.527 vs 3.383).
  • TPU7x-8 AOT & live benchmarks ([measured tpu7x-8], FSDP=8, seq=4096, remat_policy=full, Cloud Logging: https://cloudlogging.app.goo.gl/hts4JBLk5K3Ywh6EA):
    • qwen3-8b (8 layers): m3 off 182.23 ms -> prefetch 156.66 ms (1.163x) / prefetch+deferred-RS 155.57 ms (1.171x), beating legacy MaxText prefetch (173.96 / 172.11 ms) by 1.110x / 1.106x (step-0 dLoss = 3.13e-4, max dGnorm = 1.19e-7..2.38e-7).
    • qwen3-0.6b (28 layers): m3 off 128.98 ms -> prefetch 110.17 ms (1.171x), beating legacy prefetch (125.47 ms) by 1.139x.
    • [compiled AOT] tpu7x-8 / tpu7x-128: 30/30 configs pass (AG 28/28, RS 14/14, 0 exposed), saving 1.50–2.64 GiB residual HBM vs legacy prefetch; default-off normalized HLO SHA-256 is byte-identical.
    • [measured v5p-8] (FSDP=4, no SparseCore offload): 329.76 vs 329.67 ms (1.000x, neutral vs off; 1.052x–1.079x faster than legacy prefetch).
  • Pre-commit linters: pyink, pylint (10.00/10), and codespell clean.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

Inside scanned m3 decoder layers, XLA latency-hiding flags alone produce
byte-identical HLOs and leave per-layer FSDP weight all-gathers serialized or
split across windowed einsums on TPU7x. Adding double-buffered FSDP weight
prefetching hides layer k+1's weight all-gather (and optionally layer k+1's
gradient reduce-scatter) behind layer k's compute while keeping parameter
paths unchanged and leaving compiled HLO byte-identical when disabled.

- Add `src/maxtext/m3/infra/fsdp_prefetch.py`: path-transparent
  `FsdpPrefetchRunner`, `apply_fsdp_prefetch`, and `maybe_apply_fsdp_prefetch`
  implementing a `jax.custom_vjp` `unroll=2` double-buffered scan pipeline
  supporting explicit and auto meshes, multi-axis sharding (2D FSDP,
  FSDP+TP, FSDP+DP), SparseCore 1 pinning on multi-trip TPU7x scan loops
  (`num_layers >= 6`), host-stack residual streaming for `minimal_offloaded`
  remat policies, fail-closed rejection of `remat_policy='none'`, and
  safe fallback for `num_layers <= 1`, non-FSDP meshes, decode mode, and
  `parameter_memory_host_offload=True`.
- Add optional `fsdp_prefetch` hook on `ScannedLayers` in
  `src/maxtext/m3/infra/scan.py` (preserving `scan -> fsdp_prefetch` and
  `fsdp_prefetch -> scan` order invariance and zero new imports).
- Add `prefetch_fsdp_weights` and `prefetch_fsdp_defer_grad_reduce_scatter`
  (both default `False`) to `TrainerConfig` in `src/maxtext/m3/configs/config.py`,
  and wire `maybe_apply_fsdp_prefetch` into both production drivers
  (`src/maxtext/utils/model_creation_utils.py` and `src/maxtext/m3/train/loop.py`).
- Add `tests/unit/m3/fsdp_prefetch_test.py` covering helper logic, forward
  and backward parity across explicit/auto/multi-axis/stateful/minimal_offloaded
  configurations with perturbation negative controls, fail-closed checks,
  dual-driver wiring, compiled-HLO byte-identity when disabled, and compiled-HLO
  overlap structure when enabled.
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces an explicit FSDP weight prefetching mechanism for m3 ScannedLayers to address latency bottlenecks in TPU7x environments. By utilizing a double-buffered pipeline and optional deferred gradient reduce-scatter, the changes effectively overlap communication with computation. The implementation is designed to be path-transparent, maintaining compatibility with existing model structures and ensuring that performance remains unchanged when the feature is disabled.

Highlights

  • FSDP Weight Prefetching: Implemented double-buffered FSDP weight prefetching for m3 ScannedLayers, enabling overlap of weight all-gathers with layer computation to improve latency.
  • Deferred Gradient Reduce-Scatter: Added an optional hook to defer weight-gradient reduce-scatter operations to the next layer's backward step, optimizing HBM usage.
  • Infrastructure Integration: Integrated the prefetch logic into the m3 infrastructure via a path-transparent runner, ensuring byte-identical HLO output when the feature is disabled.
  • Performance and Compatibility: Added support for multi-axis sharding, SparseCore pinning on TPU7x, and host-stack residual streaming for minimal_offloaded remat policies.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 17, 2026
@AusarYao28 AusarYao28 removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 17, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: XLA latency-hiding flags alone leave FSDP weight all-gathers serialized or split across windowed einsums in m3 ScannedLayers.
Spec item Implementation Validating test
R1 (stated): Add double-buffered FSDP weight prefetching and deferred gradient reduce-scatter. ✅ tests/unit/m3/fsdp_prefetch_test.py::test_compiled_hlo_prefetches_next_layer_weight_before_current_layer_compute_ends ✅
R2 (stated): Fail closed on remat_policy='none'. ✅ tests/unit/m3/fsdp_prefetch_test.py::test_remat_none_fails_closed ✅
R3 (stated): Safe no-op fallback for non-FSDP meshes and decode mode. ✅ tests/unit/m3/fsdp_prefetch_test.py::test_non_fsdp_mesh_and_disabled_modes_fall_back_cleanly ✅
C1 (stated): Default behavior (off) must be byte-identical in compiled HLO. ✅ preserved tests/unit/m3/fsdp_prefetch_test.py::test_disabled_hlo_is_byte_identical_to_untransformed_baseline ✅
C2 (stated): Support explicit and auto meshes with multi-axis sharding. ✅ preserved tests/unit/m3/fsdp_prefetch_test.py::test_multiaxis_sharding_matches_baseline ✅

@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.95153% with 186 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/m3/infra/fsdp_prefetch.py 69.68% 134 Missing and 47 partials ⚠️
src/maxtext/m3/infra/scan.py 68.75% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant