Skip to content

Add selective pinned-host offload for large m3 embedding tables - #6406

Open
AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-host-offload-hbm-telemetryfrom
yaoyuchen/m3-qwen38-ple-host-offload
Open

AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-host-offload-hbm-telemetryfrom
yaoyuchen/m3-qwen38-ple-host-offload

Conversation

@AusarYao28

Copy link
Copy Markdown
Collaborator

Description

Depends on #6205 (stacked on yaoyuchen/m3-host-offload-hbm-telemetry; review the top commit only).

Large per-layer/n-gram embedding tables such as Qwen3.8's PLE table (~51.2B parameters, ~102 GB in bf16) plus their AdamW optimizer states consume roughly half of per-chip HBM while each step reads only B * S * ngram rows. Existing m3 host offload (parameter_memory_host_offload / optimizer_memory_host_offload) is all-or-nothing.

This PR adds generic regex-selected pinned-host embedding + AdamW offload in src/maxtext/m3/infra/offload.py, off by default and byte-identical in compiled HLO when disabled, with zero model-name branching:

  • Adds selective_host_offload (default False), host_offload_param_patterns, selective_host_offload_optimizer_mode (host_compute default exact dense AdamW, streamed_chunks, or opt-in lazy_sparse), and selective_host_offload_num_chunks in configs/base.yml, configs/types.py, and Qwen3.8 model configs (targeting the PLE n-gram table when enabled).
  • Keeps selected embedding parameters and their AdamW moments in pinned_host while leaving all other weights and optimizer states on device; gathers only step-active rows on forward and produces row-sparse gradients (SparseHostGrad) without materializing a dense HBM gradient on backward.
  • Isolates all @compute_on("device_host") operations inside jax.shard_map so single-slice TPU compilations emit 0 MegaScale host collectives, and fuses post-update host parameter squared-norm capture into the optimizer pass.
  • Wires selective offload across both m3/train/loop.py and model_creation_utils.create_model (train.py, train_compile.py, MaxTextTrainingEngine, AbstractMaxTextEngine), failing closed with ValueError on empty pattern matches, grad_accum > 1, missing pre-gather token IDs, input shape mismatch, or lazy_sparse paired with dense host gradients.
  • Step-time trade-off (documented on the config flags): this is an HBM-enablement feature for tables that otherwise OOM. Synchronous host DRAM AdamW scans under host_compute add host step-time overhead (~776 ms/step on tpu7x-8 mini; analytically ~1.2–3.2 s/step across 51.2–76.8 GB/host/step at 48L 180B on tpu7x-128).

Tests

  • CPU unit tests (4-device host mesh, serial):
    • tests/unit/m3/offload_test.py: 22 passed (fails RED at parent #6205: 7 failed; covers validation, default-off lowered & compiled HLO SHA-256 identity, sharded 2D sparse gather/grad-norm/clipping/optimizer modes, scanned & unscanned Qwen3.8-mini 3-step AdamW parity and pinned_host invariants, dual-driver & transform composition, and fail-closed checks).
    • Regression suites: train_loop_test.py + train_nnx_test.py + remat_test.py + modeling_qwen3_8_flash_next_test.py: 142 passed, 13 skipped.
    • Negative control: dropping AdamW bias correction fails 2/2 dense-vs-host parity tests.
  • Live TPU7x-8 benchmark ([measured tpu7x-8], qwen3.8-flash-next-mini, B=8, S=256, FSDP=4, EP=2, Cloud Logging: https://cloudlogging.app.goo.gl/qofwwiCbeN15kekG8):
    • Offload off: 21.22 ms/step, 1345.7 MiB peak alloc HBM
    • host_compute (default exact dense AdamW): 797.53 ms/step, 306.6 MiB peak alloc HBM (-77.2%)
    • lazy_sparse (opt-in touched-rows approx): 664.42 ms/step, 306.6 MiB peak alloc HBM (-77.2%)
  • TPU7x AOT HBM ([compiled AOT], MegaScale=0):
    • tpu7x-8 mini (ngram_vocab_size_base=200000, full remat, FSDP=4, EP=2): loop 1265.0 -> 123.1 MiB (-90.3%), engine 593.9 -> 116.6 MiB (-80.4%).
    • (Note: when combined locally with Zhijing's #6274 128-way ngram_vocab row sharding on 48L 180B tpu7x-128, AbstractMaxTextEngine peak HBM drops 31.06 -> 21.83 GiB [-29.7%]; #6274 is not included in this PR.)
  • Pre-commit linters: pyink, pylint (10.00/10), yamllint, and codespell clean.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

Qwen3.8's PLE n-gram embedding table (~51.2B params, ~102 GB bf16) plus its
AdamW state consumes roughly half of per-chip HBM state while each step reads
only B*S*ngram rows. Existing m3 offload (`parameter_memory_host_offload` /
`optimizer_memory_host_offload`) is all-or-nothing.

- Add generic path-pattern selective pinned-host offload in
  `src/maxtext/m3/infra/offload.py` (`selective_host_offload`,
  `host_offload_param_patterns`, `selective_host_offload_optimizer_mode`,
  `selective_host_offload_num_chunks`), off by default and HLO-identical when
  disabled, with Qwen3.8 model configs targeting the PLE n-gram embedding.
- Keep selected embedding parameters and their AdamW state in `pinned_host`
  while leaving all other weights and optimizer state on device; gather only
  step-active rows on forward and produce row-sparse gradients (`SparseHostGrad`)
  without materializing a dense HBM gradient on backward.
- Preserve exact dense AdamW semantics by default via `host_compute` (with
  `streamed_chunks` and documented opt-in `lazy_sparse` modes), isolate all
  host-compute ops inside `jax.shard_map` so single-slice TPU compilations emit
  zero MegaScale host collectives, and fuse post-update host param norm capture
  into the optimizer pass to avoid extra HBM spills.
- Wire selective offload across both `m3/train/loop.py` and
  `model_creation_utils.create_model` (`train.py`, `train_compile.py`,
  `MaxTextTrainingEngine`, `AbstractMaxTextEngine`) while preserving param paths
  and `scan -> remat -> quantize -> named scopes -> offload -> optimizer` order.
- Add `SelectiveHostOffloadTest` in `tests/unit/m3/offload_test.py` covering
  validation, default-off HLO identity, sharded 2D sparse gather/grad-norm/
  clipping/optimizer modes, scanned and unscanned Qwen3.8-mini 3-step parity and
  `pinned_host` memory_kind invariants, dual-driver and transform composition,
  and TPU7x-8 AOT HBM savings with zero MegaScale collectives.
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a selective host offload feature designed to mitigate HBM memory pressure caused by large embedding tables, such as those found in Qwen3.8 models. By offloading only specific regex-matched parameters and their optimizer states to pinned host memory, the model can accommodate significantly larger embedding tables without exceeding device memory limits. The implementation includes specialized row-sparse gradient handling and multiple optimizer update strategies to balance memory savings with step-time performance.

Highlights

  • Selective Host Offload: Introduced a generic, regex-based selective host offload mechanism for embedding tables and their AdamW optimizer states, allowing them to reside in pinned host memory while keeping other weights on device.
  • Sparse Gradient Support: Implemented SparseHostGrad to support row-sparse gradient updates, avoiding the need to materialize dense full-table gradients in HBM during backward passes.
  • Optimizer Modes: Added three optimizer modes for offloaded tables: 'host_compute' (default exact dense AdamW on host), 'streamed_chunks' (for memory-constrained scenarios), and 'lazy_sparse' (approximate update for touched rows).
  • Infrastructure Integration: Integrated selective offload across the training loop and engine, including support for sharded 2D embedding tables, global norm clipping, and automatic fail-closed validation.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@AusarYao28 AusarYao28 removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 17, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec partially met; streamed_chunks emits MegaScale collectives.

Top issues:

  1. 🟠 [Perf] _selective_streamed_chunks_update uses a global jax.lax.dynamic_slice_in_dim on p_h (a pinned_host array). If this array is row-sharded, slicing it outside of jax.shard_map forces XLA to emit MegaScale host collectives to communicate the slices across hosts, violating the zero-collective constraint and explaining the ~150x slowdown on live TPUs. → Wrap the sliced streamed updates inside jax.shard_map like _host_embed_gather, or update the description/docs to explicitly state that streamed_chunks emits MegaScale host collectives by design.
  2. 🟠 [Alignment] See inline comment on src/maxtext/configs/types.py. Config validation for selective_host_offload is currently placed inside the unrelated validate_save_fsdp_gathered_weights validator. → Move the new validation to a dedicated @model_validator(mode="after") named validate_selective_host_offload.
Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Large embedding tables (like Qwen3.8 PLE) and their AdamW moments consume too much HBM (~102 GB), and existing host offload is all-or-nothing, imposing high latency if applied to everything.
Spec item Implementation Validating test
R1 (stated): Allow pattern-based offloading of specific parameter tables and their moments to pinned_host. ✅ test_qwen38_mini_loop_parity_and_memory_kind ✅
R2 (stated): Gather step-active rows dynamically and produce row-sparse gradients without dense HBM materialization. ✅ test_tpu7x_aot_selective_ple_offload_reduces_hbm_without_dense_grad ✅
R3 (stated): Fail closed on empty patterns, grad_accum > 1, missing pre-gather token IDs, shape mismatch, or lazy_sparse + dense grads. ✅ test_selective_offload_fails_closed_instead_of_silent_fallback ✅
C1 (stated): Isolate operations so single-slice TPU compilations emit 0 MegaScale host collectives. ⚠️ partial test_tpu7x_aot_selective_ple_offload_reduces_hbm_without_dense_grad ⚠️ superficial (only tests host_compute which is collective-free, missing coverage for streamed_chunks which emits them)
C2 (stated): Feature is off by default and yields byte-identical HLO when disabled. ✅ preserved test_default_off_hlo_byte_identical ✅

Comment on lines +5507 to +5511
if self.selective_host_offload:
if self.parameter_memory_host_offload:
raise ValueError("selective_host_offload=True cannot be combined with parameter_memory_host_offload=True.")
if not self.host_offload_param_patterns:
raise ValueError("selective_host_offload=True requires non-empty host_offload_param_patterns.")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Fixes at the wrong layer: this config validation for selective_host_offload is placed inside validate_save_fsdp_gathered_weights. Create a dedicated @model_validator(mode="after") named validate_selective_host_offload to keep unrelated concerns separate.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant