Repository navigation
Add explicit FSDP weight prefetching for m3 ScannedLayers - #6405
AusarYao28 wants to merge 1 commit into
Conversation
Inside scanned m3 decoder layers, XLA latency-hiding flags alone produce byte-identical HLOs and leave per-layer FSDP weight all-gathers serialized or split across windowed einsums on TPU7x. Adding double-buffered FSDP weight prefetching hides layer k+1's weight all-gather (and optionally layer k+1's gradient reduce-scatter) behind layer k's compute while keeping parameter paths unchanged and leaving compiled HLO byte-identical when disabled. - Add `src/maxtext/m3/infra/fsdp_prefetch.py`: path-transparent `FsdpPrefetchRunner`, `apply_fsdp_prefetch`, and `maybe_apply_fsdp_prefetch` implementing a `jax.custom_vjp` `unroll=2` double-buffered scan pipeline supporting explicit and auto meshes, multi-axis sharding (2D FSDP, FSDP+TP, FSDP+DP), SparseCore 1 pinning on multi-trip TPU7x scan loops (`num_layers >= 6`), host-stack residual streaming for `minimal_offloaded` remat policies, fail-closed rejection of `remat_policy='none'`, and safe fallback for `num_layers <= 1`, non-FSDP meshes, decode mode, and `parameter_memory_host_offload=True`. - Add optional `fsdp_prefetch` hook on `ScannedLayers` in `src/maxtext/m3/infra/scan.py` (preserving `scan -> fsdp_prefetch` and `fsdp_prefetch -> scan` order invariance and zero new imports). - Add `prefetch_fsdp_weights` and `prefetch_fsdp_defer_grad_reduce_scatter` (both default `False`) to `TrainerConfig` in `src/maxtext/m3/configs/config.py`, and wire `maybe_apply_fsdp_prefetch` into both production drivers (`src/maxtext/utils/model_creation_utils.py` and `src/maxtext/m3/train/loop.py`). - Add `tests/unit/m3/fsdp_prefetch_test.py` covering helper logic, forward and backward parity across explicit/auto/multi-axis/stateful/minimal_offloaded configurations with perturbation negative controls, fail-closed checks, dual-driver wiring, compiled-HLO byte-identity when disabled, and compiled-HLO overlap structure when enabled.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces an explicit FSDP weight prefetching mechanism for m3 ScannedLayers to address latency bottlenecks in TPU7x environments. By utilizing a double-buffered pipeline and optional deferred gradient reduce-scatter, the changes effectively overlap communication with computation. The implementation is designed to be path-transparent, maintaining compatibility with existing model structures and ensuring that performance remains unchanged when the feature is disabled. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: XLA latency-hiding flags alone leave FSDP weight all-gathers serialized or split across windowed einsums in m3
ScannedLayers.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Add double-buffered FSDP weight prefetching and deferred gradient reduce-scatter. | ✅ | tests/unit/m3/fsdp_prefetch_test.py::test_compiled_hlo_prefetches_next_layer_weight_before_current_layer_compute_ends ✅ |
R2 (stated): Fail closed on remat_policy='none'. |
✅ | tests/unit/m3/fsdp_prefetch_test.py::test_remat_none_fails_closed ✅ |
| R3 (stated): Safe no-op fallback for non-FSDP meshes and decode mode. | ✅ | tests/unit/m3/fsdp_prefetch_test.py::test_non_fsdp_mesh_and_disabled_modes_fall_back_cleanly ✅ |
| C1 (stated): Default behavior (off) must be byte-identical in compiled HLO. | ✅ preserved | tests/unit/m3/fsdp_prefetch_test.py::test_disabled_hlo_is_byte_identical_to_untransformed_baseline ✅ |
| C2 (stated): Support explicit and auto meshes with multi-axis sharding. | ✅ preserved | tests/unit/m3/fsdp_prefetch_test.py::test_multiaxis_sharding_matches_baseline ✅ |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Description
Depends on #6249 (stacked on
yaoyuchen/m3-remat-hybrid-barriers; review the top commit only).Inside scanned
m3decoder layers (ScannedLayers), XLA latency-hiding flags alone produce byte-identical HLOs on TPU7x and leave per-layer FSDP weightall-gathers serialized or split across windowed einsums. This PR adds explicit double-buffered FSDP weight prefetching (and optional deferred gradientreduce-scatter) form3ScannedLayers, off by default and byte-identical in compiled HLO when disabled.src/maxtext/m3/infra/fsdp_prefetch.py(FsdpPrefetchRunner,apply_fsdp_prefetch,maybe_apply_fsdp_prefetch) implementing a path-transparentjax.custom_vjpunroll=2double-buffered scan pipeline supportingautoandexplicitmeshes, multi-axis sharding (2D FSDP, FSDP+TP, FSDP+DP), SparseCore 1 pinning on multi-trip TPU7x scan loops (num_layers >= 6, avoiding the TPU7xComputeEmitterall-gather(%convert)crash on inlined<6-layer loops), host-stack residual streaming forminimal_offloadedremat policies (failing closed on residual leaf-count mismatch), fail-closed rejection ofremat_policy='none', and safe no-op fallback fornum_layers <= 1, non-FSDP meshes, decode mode, andparameter_memory_host_offload=True.fsdp_prefetchhook onScannedLayersinsrc/maxtext/m3/infra/scan.py(order-invariant withscanand zero new imports inscan.py). Qwen3.8 cycle scans are intentionally untouched here and will follow after#6325.prefetch_fsdp_weightsandprefetch_fsdp_defer_grad_reduce_scatter(both defaultFalse) toTrainerConfiginsrc/maxtext/m3/configs/config.py, wired into bothsrc/maxtext/utils/model_creation_utils.pyandsrc/maxtext/m3/train/loop.py.minimal_offloadedwithprefetch_fsdp_weights=Truecompiles and matches numerics, but synchronous host-stackdynamic_update_slicestalls PCIe DMA on TPU7x (815 vs 244 ms/stepontpu7x-8).Tests
tests/unit/m3/fsdp_prefetch_test.py: 24 passed (covers forward/backward parity across explicit, auto, multi-axis, stateful, andminimal_offloadedconfigs, fail-closed checks, dual-driver wiring, compiled-HLO byte-identity when disabled, and overlap HLO structure when enabled).scan_tests.py(7 passed),train_loop_test.py(10 passed),m3_smoke_test.py(3 passed),model_sharding_test.py(76 passed, 3 skipped),remat_test.py(29 passed).0instead of layer1in the prefetch carry failsexplicit_full_pf_drs(loss 4.527 vs 3.383).[measured tpu7x-8],FSDP=8,seq=4096,remat_policy=full, Cloud Logging: https://cloudlogging.app.goo.gl/hts4JBLk5K3Ywh6EA):qwen3-8b(8 layers):m3off182.23 ms-> prefetch156.66 ms(1.163x) / prefetch+deferred-RS155.57 ms(1.171x), beating legacy MaxText prefetch (173.96 / 172.11 ms) by1.110x / 1.106x(step-0 dLoss = 3.13e-4,max dGnorm = 1.19e-7..2.38e-7).qwen3-0.6b(28 layers):m3off128.98 ms-> prefetch110.17 ms(1.171x), beating legacy prefetch (125.47 ms) by1.139x.[compiled AOT]tpu7x-8/tpu7x-128:30/30configs pass (AG 28/28,RS 14/14,0exposed), saving1.50–2.64 GiBresidual HBM vs legacy prefetch; default-off normalized HLO SHA-256 is byte-identical.[measured v5p-8](FSDP=4, no SparseCore offload):329.76 vs 329.67 ms(1.000x, neutral vs off;1.052x–1.079xfaster than legacy prefetch).pyink,pylint(10.00/10), andcodespellclean.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.