Repository navigation
Add selective pinned-host offload for large m3 embedding tables - #6406
AusarYao28 wants to merge 1 commit into
Conversation
Qwen3.8's PLE n-gram embedding table (~51.2B params, ~102 GB bf16) plus its AdamW state consumes roughly half of per-chip HBM state while each step reads only B*S*ngram rows. Existing m3 offload (`parameter_memory_host_offload` / `optimizer_memory_host_offload`) is all-or-nothing. - Add generic path-pattern selective pinned-host offload in `src/maxtext/m3/infra/offload.py` (`selective_host_offload`, `host_offload_param_patterns`, `selective_host_offload_optimizer_mode`, `selective_host_offload_num_chunks`), off by default and HLO-identical when disabled, with Qwen3.8 model configs targeting the PLE n-gram embedding. - Keep selected embedding parameters and their AdamW state in `pinned_host` while leaving all other weights and optimizer state on device; gather only step-active rows on forward and produce row-sparse gradients (`SparseHostGrad`) without materializing a dense HBM gradient on backward. - Preserve exact dense AdamW semantics by default via `host_compute` (with `streamed_chunks` and documented opt-in `lazy_sparse` modes), isolate all host-compute ops inside `jax.shard_map` so single-slice TPU compilations emit zero MegaScale host collectives, and fuse post-update host param norm capture into the optimizer pass to avoid extra HBM spills. - Wire selective offload across both `m3/train/loop.py` and `model_creation_utils.create_model` (`train.py`, `train_compile.py`, `MaxTextTrainingEngine`, `AbstractMaxTextEngine`) while preserving param paths and `scan -> remat -> quantize -> named scopes -> offload -> optimizer` order. - Add `SelectiveHostOffloadTest` in `tests/unit/m3/offload_test.py` covering validation, default-off HLO identity, sharded 2D sparse gather/grad-norm/ clipping/optimizer modes, scanned and unscanned Qwen3.8-mini 3-step parity and `pinned_host` memory_kind invariants, dual-driver and transform composition, and TPU7x-8 AOT HBM savings with zero MegaScale collectives.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements a selective host offload feature designed to mitigate HBM memory pressure caused by large embedding tables, such as those found in Qwen3.8 models. By offloading only specific regex-matched parameters and their optimizer states to pinned host memory, the model can accommodate significantly larger embedding tables without exceeding device memory limits. The implementation includes specialized row-sparse gradient handling and multiple optimizer update strategies to balance memory savings with step-time performance. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec partially met; streamed_chunks emits MegaScale collectives.
Top issues:
- 🟠 [Perf]
_selective_streamed_chunks_updateuses a globaljax.lax.dynamic_slice_in_dimonp_h(apinned_hostarray). If this array is row-sharded, slicing it outside ofjax.shard_mapforces XLA to emit MegaScale host collectives to communicate the slices across hosts, violating the zero-collective constraint and explaining the ~150x slowdown on live TPUs. → Wrap the sliced streamed updates insidejax.shard_maplike_host_embed_gather, or update the description/docs to explicitly state thatstreamed_chunksemits MegaScale host collectives by design. - 🟠 [Alignment] See inline comment on
src/maxtext/configs/types.py. Config validation forselective_host_offloadis currently placed inside the unrelatedvalidate_save_fsdp_gathered_weightsvalidator. → Move the new validation to a dedicated@model_validator(mode="after")namedvalidate_selective_host_offload.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: Large embedding tables (like Qwen3.8 PLE) and their AdamW moments consume too much HBM (~102 GB), and existing host offload is all-or-nothing, imposing high latency if applied to everything.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Allow pattern-based offloading of specific parameter tables and their moments to pinned_host. |
✅ | test_qwen38_mini_loop_parity_and_memory_kind ✅ |
| R2 (stated): Gather step-active rows dynamically and produce row-sparse gradients without dense HBM materialization. | ✅ | test_tpu7x_aot_selective_ple_offload_reduces_hbm_without_dense_grad ✅ |
| R3 (stated): Fail closed on empty patterns, grad_accum > 1, missing pre-gather token IDs, shape mismatch, or lazy_sparse + dense grads. | ✅ | test_selective_offload_fails_closed_instead_of_silent_fallback ✅ |
| C1 (stated): Isolate operations so single-slice TPU compilations emit 0 MegaScale host collectives. | test_tpu7x_aot_selective_ple_offload_reduces_hbm_without_dense_grad host_compute which is collective-free, missing coverage for streamed_chunks which emits them) |
|
| C2 (stated): Feature is off by default and yields byte-identical HLO when disabled. | ✅ preserved | test_default_off_hlo_byte_identical ✅ |
| if self.selective_host_offload: | ||
| if self.parameter_memory_host_offload: | ||
| raise ValueError("selective_host_offload=True cannot be combined with parameter_memory_host_offload=True.") | ||
| if not self.host_offload_param_patterns: | ||
| raise ValueError("selective_host_offload=True requires non-empty host_offload_param_patterns.") |
There was a problem hiding this comment.
🟠 Fixes at the wrong layer: this config validation for selective_host_offload is placed inside validate_save_fsdp_gathered_weights. Create a dedicated @model_validator(mode="after") named validate_selective_host_offload to keep unrelated concerns separate.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Description
Depends on #6205 (stacked on
yaoyuchen/m3-host-offload-hbm-telemetry; review the top commit only).Large per-layer/n-gram embedding tables such as Qwen3.8's PLE table (~51.2B parameters, ~102 GB in
bf16) plus their AdamW optimizer states consume roughly half of per-chip HBM while each step reads onlyB * S * ngramrows. Existingm3host offload (parameter_memory_host_offload/optimizer_memory_host_offload) is all-or-nothing.This PR adds generic regex-selected pinned-host embedding + AdamW offload in
src/maxtext/m3/infra/offload.py, off by default and byte-identical in compiled HLO when disabled, with zero model-name branching:selective_host_offload(defaultFalse),host_offload_param_patterns,selective_host_offload_optimizer_mode(host_computedefault exact dense AdamW,streamed_chunks, or opt-inlazy_sparse), andselective_host_offload_num_chunksinconfigs/base.yml,configs/types.py, and Qwen3.8 model configs (targeting the PLE n-gram table when enabled).pinned_hostwhile leaving all other weights and optimizer states on device; gathers only step-active rows on forward and produces row-sparse gradients (SparseHostGrad) without materializing a dense HBM gradient on backward.@compute_on("device_host")operations insidejax.shard_mapso single-slice TPU compilations emit0MegaScale host collectives, and fuses post-update host parameter squared-norm capture into the optimizer pass.m3/train/loop.pyandmodel_creation_utils.create_model(train.py,train_compile.py,MaxTextTrainingEngine,AbstractMaxTextEngine), failing closed withValueErroron empty pattern matches,grad_accum > 1, missing pre-gather token IDs, input shape mismatch, orlazy_sparsepaired with dense host gradients.host_computeadd host step-time overhead (~776 ms/stepontpu7x-8mini; analytically~1.2–3.2 s/stepacross51.2–76.8 GB/host/stepat 48L 180B ontpu7x-128).Tests
tests/unit/m3/offload_test.py: 22 passed (fails RED at parent#6205:7 failed; covers validation, default-off lowered & compiled HLO SHA-256 identity, sharded 2D sparse gather/grad-norm/clipping/optimizer modes, scanned & unscanned Qwen3.8-mini 3-step AdamW parity andpinned_hostinvariants, dual-driver & transform composition, and fail-closed checks).train_loop_test.py+train_nnx_test.py+remat_test.py+modeling_qwen3_8_flash_next_test.py: 142 passed, 13 skipped.2/2dense-vs-host parity tests.[measured tpu7x-8],qwen3.8-flash-next-mini,B=8, S=256, FSDP=4, EP=2, Cloud Logging: https://cloudlogging.app.goo.gl/qofwwiCbeN15kekG8):21.22 ms/step,1345.7 MiBpeak alloc HBMhost_compute(default exact dense AdamW):797.53 ms/step,306.6 MiBpeak alloc HBM (-77.2%)lazy_sparse(opt-in touched-rows approx):664.42 ms/step,306.6 MiBpeak alloc HBM (-77.2%)[compiled AOT],MegaScale=0):tpu7x-8mini (ngram_vocab_size_base=200000,fullremat,FSDP=4, EP=2): loop1265.0 -> 123.1 MiB(-90.3%), engine593.9 -> 116.6 MiB(-80.4%).#6274128-wayngram_vocabrow sharding on 48L 180Btpu7x-128,AbstractMaxTextEnginepeak HBM drops31.06 -> 21.83 GiB[-29.7%];#6274is not included in this PR.)pyink,pylint(10.00/10),yamllint, andcodespellclean.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.