Repository navigation
feat(m3): adopt shared MoEExecutor for GraniteMoE and multi-rank EP GLM-5.3 - #6282
AusarYao28 wants to merge 1 commit into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request adopts a unified Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
|
🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details. |
f93f3f9 to
ee2aff7
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details. |
ee2aff7 to
0beb5c1
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec partially met; TPU tail-row masking incomplete and multi-host device binding bug.
Top issues:
⚠️ [Description] PR description omits the change to_validate_generic_maxtext_kwargsinmodeling_glm5_3_flash.py, which adds call-stack frame inspection forloss_fn. → Document why_validate_generic_maxtext_kwargswas modified, or preferably handle trainer kwargs explicitly without reflection.- 🔴 [Bug] Unassigned tail rows in
_pallas_grouped_matmuland_pallas_grouped_matmul_bwdare masked on CPU, butlhsremains unmasked on TPU before padding forgmm_v2andtgmm_v2. If unassigned tail rows containNaNs, the contraction intgmm_v2will propagateNaNinto weight gradientdrhs; see inline comments onsrc/maxtext/m3/core/moe.py. → Masklhswithvalid_rowsbefore padding on TPU in both forward and backward passes. - [Alignment]
_validate_generic_maxtext_kwargsusesinspect.currentframe()to dynamically walk caller frames and detectloss_fninmaxtext/trainers. This reflection is fragile in JAX tracing and inconsistent with sibling M3 models; see inline comment onmodeling_glm5_3_flash.py. → Handle or ignore generic trainer kwargs (enable_dropout,decoder_target_tokens) directly at the model interface without inspecting frames. - [Bug]
grouped_unsharded_fusedbuilds a 1-device mesh usingjax.devices()[:1], which crashes on multi-host TPU slices because worker hosts > 0 do not own device 0; see inline comment onsrc/maxtext/m3/core/moe.py. → Usejax.local_devices()[:1]. - [Alignment]
GraniteMoeExpertsbranches onbackend in ("tokamax", "megablox")and forwardsbackendtom3_moe.grouped_matmul, which rejects"megablox"withValueError; see inline comment onmodeling_granitemoe.py. → Standardize onbackend == "tokamax".
Inferred spec & validation
- Sources: PR description; linked Buganizer ticket b/580120934 (not readable here); code before/after.
- Problem: GraniteMoe and GLM-5.3-Flash lacked unified
MoEExecutoradoption, leaving GLM-5.3-Flash without multi-device EP (ep_size > 1) support and exposing potential tail-row padding and alignment hazards on grouped matmuls.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): GraniteMoeConfig and Glm5Config support sparse_matmul: bool = True with strict boolean validation |
✅ met | glm5_3_flash_config_test.py::test_sparse_matmul_validation_and_sharding_axes ✅, granitemoe_test.py::test_sparse_matmul_config_validation_and_sharding_axes ✅ |
R2 (stated): Enable multi-device EP (ep_size > 1) on Glm5MoE via MoEExecutor.fused without expert-weight all-gathers |
✅ met | glm5_3_flash_moe_test.py::test_multidevice_tp_and_ep_tokamax_execution_and_no_weight_all_gather ✅ |
R3 (stated): Preserve Glm5Experts native ragged_dot execution path when ep_size == 1 |
✅ met | glm5_3_flash_moe_test.py::test_single_device_and_small_token_tokamax_forward_and_vjp_parity ✅ |
R4 (stated): Zero-mask unassigned tail rows (row_idx >= sum(group_sizes)) across ragged_dot and tokamax forward and backward |
moe_test.py::test_grouped_matmul_unassigned_tail_rows_zeroed_in_fwd_and_bwd is_tpu=False, masking unmasked TPU lhs_pad) |
|
R5 (stated): Wrap grouped_unsharded_fused in a 1-device shard_map on TPU when no mesh is in scope |
moe_test.py::test_grouped_unsharded_fused_without_active_mesh jax.devices()[:1]) |
|
C1 (stated): Backward compatibility and no performance regression for ep_size == 1 |
✅ preserved | glm5_3_flash_moe_test.py::test_ep_sharding_guard ✅ |
C2 (stated): Compound ("fsdp", "expert") weight sharding in GraniteMoe preserved with 0 expert all-gathers |
✅ preserved | granitemoe_test.py::test_single_and_multidevice_tokamax_forward_vjp_parity_and_no_weight_all_gather ✅ |
| C3 (inferred): Standard trainer interface kwargs handled without runtime stack inspection | ❌ violated | ❌ none |
| if not is_tpu: | ||
| return jax.lax.ragged_dot( | ||
| lhs, | ||
| lhs_safe = jnp.where(valid_rows, lhs, jnp.zeros((), dtype=lhs.dtype)) |
There was a problem hiding this comment.
🔴 lhs_safe is computed inside if not is_tpu: and only used for the CPU ragged_dot fallback. On TPU (line 187), lhs_pad = jnp.pad(lhs, ...) pads the unmasked lhs, meaning unassigned tail rows (row_idx >= valid_count) containing garbage or NaNs are passed directly into gmm_v2. Compute lhs_safe = jnp.where(valid_rows, lhs, jnp.zeros((), dtype=lhs.dtype)) before if not is_tpu: and pad lhs_safe on TPU.
There was a problem hiding this comment.
Fixed in 752e2bf4af (and parent #6164 53c481bad3): hoisted lhs_safe = jnp.where(valid_rows, lhs, 0) above if not is_tpu: in _pallas_grouped_matmul and grouped_matmul so TPU gmm_v2 / tokamax.ragged_dot paths zero unassigned tail rows before padding.
|
|
||
| if backend_kind == "tokamax": | ||
| if not is_tpu: | ||
| lhs_safe = jnp.where(valid_rows, lhs, jnp.zeros((), dtype=lhs.dtype)) |
There was a problem hiding this comment.
🔴 lhs_safe in the backward pass is only defined within if not is_tpu:. On TPU (line 252), lhs_pad = jnp.pad(lhs, ...) passes the unmasked lhs to tgmm_v2.tgmm_v2(lhs=lhs_pad, rhs=grad_pad). If unassigned tail rows in lhs contain NaNs, the contraction with grad_pad will propagate NaN into weight gradients (drhs). Define lhs_safe outside if not is_tpu: and pass lhs_safe to lhs_pad.
There was a problem hiding this comment.
Fixed in 752e2bf4af (and parent #6164 53c481bad3): hoisted lhs_safe = jnp.where(valid_rows, lhs, 0) above if not is_tpu: in _pallas_grouped_matmul_bwd and passed lhs_safe into lhs_pad = jnp.pad(lhs_safe, ...) before tgmm_v2.tgmm_v2 on TPU.
| projections = (gate_up_proj.astype(x.dtype), down_proj.astype(x.dtype)) | ||
| backend = executor.kind if executor.kind in ("ragged_dot", "tokamax") else "ragged_dot" | ||
| if is_tpu and backend == "tokamax": | ||
| mesh_1d = jax.sharding.Mesh(jax.devices()[:1], ("_moe_1d",)) |
There was a problem hiding this comment.
🟠 On multi-host TPU slices (e.g. v4-16, v5p-128), jax.devices()[:1] selects device 0 on worker host 0. Any non-zero worker host will crash with a device locality error because device 0 is not owned by that process. Use jax.local_devices()[:1] so each host binds to its local TPU device.
| mesh_1d = jax.sharding.Mesh(jax.devices()[:1], ("_moe_1d",)) | |
| mesh_1d = jax.sharding.Mesh(jax.local_devices()[:1], ("_moe_1d",)) |
There was a problem hiding this comment.
Fixed in 752e2bf4af: grouped_unsharded_fused now builds its single-device mesh from jax.local_devices()[:1] rather than jax.devices()[:1] so multi-host TPU processes only reference a host-local addressable device.
| precision=precision, | ||
| ) | ||
| dn = down_w.astype(dt) | ||
| if backend in ("tokamax", "megablox"): |
There was a problem hiding this comment.
🟠 m3_moe.grouped_matmul only accepts backend="ragged_dot" or backend="tokamax", raising a ValueError if passed "megablox". In addition, MoEExecutor.kind is typed as Literal["dense", "ragged_dot", "tokamax"]. Standardize on backend == "tokamax" here and in lines 774, 784, 799, and 803.
| if backend in ("tokamax", "megablox"): | |
| if backend == "tokamax": |
There was a problem hiding this comment.
Fixed in 752e2bf4af (with SHARDED_EXECUTOR_KINDS = ('gmm_v2', 'tokamax') in #6164 53c481bad3): GraniteMoeExperts and Glm5MoE now resolve executor kinds via moe_from_config ('gmm_v2' for megablox=True, 'tokamax' for use_tokamax_gmm=True) and check backend in m3_moe.SHARDED_EXECUTOR_KINDS everywhere instead of passing 'megablox' to grouped_matmul.
| import inspect | ||
|
|
||
| is_generic_trainer = False | ||
| frame = inspect.currentframe() | ||
| while frame is not None: | ||
| if frame.f_code.co_name == "loss_fn" and "maxtext/trainers" in frame.f_code.co_filename: | ||
| is_generic_trainer = True | ||
| break | ||
| frame = frame.f_back | ||
|
|
There was a problem hiding this comment.
🟠 Inspecting the runtime call stack via inspect.currentframe() to check for loss_fn in maxtext/trainers is fragile under JIT tracing, function decorators, or custom trainer/evaluation loops. Consistent with sibling M3 models (such as GraniteMoe and Qwen3), handle or ignore generic trainer kwargs (enable_dropout, decoder_target_tokens) directly at the layer interface instead of inspecting stack frames.
There was a problem hiding this comment.
Fixed in 752e2bf4af: removed the inspect.currentframe() stack inspection and restored Glm5Model._validate_generic_maxtext_kwargs to its fail-closed origin/main behavior (verified by test_generic_maxtext_kwargs_validation).
| ref_dlhs, ref_drhs = ref_vjp(cot_valid) | ||
|
|
||
| out, vjp_fn = jax.vjp( | ||
| lambda l, r: m3_moe.grouped_matmul(l, r, group_sizes, backend=backend, is_tpu=False), |
There was a problem hiding this comment.
🟡 This test specifies is_tpu=False for both parameterized backends, which routes "tokamax" through the CPU ragged_dot fallback and leaves the TPU Pallas kernels (gmm_v2 / tgmm_v2) and padding logic unexercised. Add a TPU test (or test double) to verify that unassigned tail rows are zeroed in lhs_pad on TPU.
There was a problem hiding this comment.
Fixed in 752e2bf4af: added a mocked-TPU (is_tpu=True) test injecting NaNs into unassigned tail rows [sum(group_sizes):M] and verifying gmm_v2.gmm_v2 and tgmm_v2.tgmm_v2 receive zero-masked lhs_pad/grad_pad with finite zeroed tail outputs and gradients (plus live 4-chip TPU v5p hardware execution).
2f414d9 to
53c481b
Compare
0beb5c1 to
752e2bf
Compare
53c481b to
91ef25a
Compare
752e2bf to
3bd58c0
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details. |
91ef25a to
70f09c5
Compare
3bd58c0 to
15dc9bb
Compare
70f09c5 to
e60cc45
Compare
15dc9bb to
789e1d8
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details. |
…LM-5.3
- Route GraniteMoeExperts compound ('fsdp', 'expert') and non-compound paths
through shared m3_moe.grouped_matmul and MoEExecutor.fused when
sparse_matmul=True, preserving zero expert-weight all-gather.
- Route Glm5Experts and Glm5MoE through shared MoEExecutor.fused when
ep_size > 1 and sparse_matmul=True (keeping routing weights in float32 and
sharding batch activations across ('data', 'fsdp', 'expert')), while keeping
ep_size == 1 on native ragged_dot.
- Mask unassigned tail rows [sum(group_sizes):M] in forward and backward across
m3_moe.grouped_matmul backends and add grouped_unsharded_fused fallback when
no Mesh is in scope.
FIXES: b/580120934
e60cc45 to
3d25e26
Compare
789e1d8 to
4d7dd82
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details. |
feat(m3): adopt shared MoEExecutor for GraniteMoE and multi-rank EP GLM-5.3
Description
Depends on #6164 (stack: #6066 -> #6152 -> #6163 -> #6164 -> this PR); review the top commit only.
Adopts the shared
m3MoEExecutor(src/maxtext/m3/core/moe.py, selectingkind="gmm_v2"for configs withoutuse_tokamax_gmm=Trueormegablox=False) forGraniteMoeExperts(src/maxtext/m3/models/granitemoe/modeling_granitemoe.py) and for multi-device Expert Parallelism (ep_size > 1) onGlm5Experts(src/maxtext/m3/models/glm5_3_flash/modeling_glm5_3_flash.py), usingsparse_matmul: bool = True:GraniteMoeConfig&GraniteMoeExperts(src/maxtext/m3/models/granitemoe/modeling_granitemoe.py):isinstance(sparse_matmul, bool)before coercion onGraniteMoeConfig.("fsdp", "expert")jax.shard_mappath (_ep_tp_kernel) and non-compound / single-device paths throughm3_moe.grouped_matmul/MoEExecutor.fused/grouped_unsharded_fusedwhensparse_matmul=True(padding non-divisibleep_size > 1decode batches with-1indices and slicing back), preserving compound("fsdp", "expert")expert-weight sharding (0expert-weightall-gathers) and cuttingtpu7x-8step time by1.08x–1.18x(1.21x–1.49xon 2-layer training sweeps).Glm5Config,Glm5Sharding, &Glm5Experts(src/maxtext/m3/configs/glm5_3_flash_config.py,src/maxtext/m3/models/glm5_3_flash/modeling_glm5_3_flash.py):sparse_matmul: bool = TruetoGlm5Configwith strictisinstance(sparse_matmul, bool)validation before coercion andfrom_maxtext_configforwarding.ep_size > 1) onGlm5MoE(previously raisedNotImplementedError) by dispatchingep_size > 1throughMoEExecutor.fused(jax.lax.ragged_all_to_all+ Pallas Mosaicgmm_v2/tgmm_v2, padding non-divisible batches toep_sizewith-1indices) and folding"expert"into batch activation sharding inGlm5Sharding.from_mesh/Glm5Sharding.map_axis(3.73 GiBPeak HBM and29.14 msstep time onfsdp=2, expert=4).Glm5Experts's nativejax.lax.ragged_dotpath whenep_size == 1(0%regression atEP = 1) and keepstopk_weightsinjnp.float32.src/maxtext/m3/core/moe.pyEnhancements:grouped_unsharded_fused(wrapping_grouped_unshardedin a 1-devicejax.shard_mapoverjax.local_devices()[:1]whenis_tpu=TrueacrossSHARDED_EXECUTOR_KINDS) and threadsbackend/is_tpuinto_grouped_unshardedso non-mesh callers use the resolved Pallasgmm_v2backend without executing Pallas kernels outsidejax.shard_map(the underlyinggmm_v2/tgmm_v2dispatch,(256, 128)tile alignment, and forward/TPU-bwdvalid_rowsmasking landed in#6164; live TPU"tokamax"dispatches totokamax.ragged_dot).FIXES: b/580120934
Tests
1. Targeted 4-CPU Unit & Multi-Device Parity Tests (
4-CPU Host Mesh)tests/unit/m3/granitemoe_test.py -k "multidevice":1 passed, 20 deselected(test_single_and_multidevice_tokamax_forward_vjp_parity_and_no_weight_all_gatherondata=4,fsdp=2, expert=2,fsdp=2, tensor=2, plus non-compoundEP > 1num_local_experts=6onfsdp=2, expert=2and(1, 1, H)decode batch, verifying all-leaf parameter gradient parity and zero full-expertall-gatherin compiled HLO with live positive control).tests/unit/m3/glm5_3_flash_moe_test.py -k "multidevice":1 passed, 10 deselected(test_multidevice_tp_and_ep_tokamax_execution_and_no_weight_all_gatheronfsdp=2, tensor=2,fsdp=2, expert=2,expert=4,expert=2, tensor=2, plus(1, 1, H)decode batch, verifying all-leaf parameter gradient parity and zero full-expertall-gatherin compiled HLO with live positive control).tests/unit/m3/moe_test.py -k "unassigned_tail_rows":2 passed, 34 deselected(bothtest_grouped_matmul_unassigned_tail_rows_zeroed_in_fwd_and_bwd_ragged_dotandtest_grouped_matmul_unassigned_tail_rows_zeroed_in_fwd_and_bwd_tokamaxpass).tests/unit/m3/(4-CPU host mesh, serial):509 passed, 70 skipped, 46 subtests passed.2. CPU-Hosted TPU7x AOT (
tpu7x-8andtpu7x-128) & Livetpu7x-8(2x2x1) Benchmarksbodaborg-tpu7x-nap2x2x1, 8 TensorCore devices):T_dev in {2048, 8192}): https://cloudlogging.app.goo.gl/U43TdjvmN65Zz9v29T in {16, 64, 128, 256}): https://cloudlogging.app.goo.gl/cK2UjrXoNuwRGFL6Abfloat16)tpu7x-8Mesh(B, S)(T_step)%gmm_v2/%tgmm_v2(RaggedDot->Shared MoEExecutor)RaggedDot->Shared MoEExecutor)RaggedDot->Shared MoEExecutor, GiB)p50(ms)p50(ms)granite-3.0-1b-a400m(24L)fsdp=8(8, 8)(64)0/0->24/1236->00.5566->0.55941.74->1.663.81->3.501.09x0.00e+002.54e-03granite-3.0-1b-a400m(24L)fsdp=8(8, 256)(2048)0/0->24/1236->01.2134->1.25734.33->4.2111.02->9.511.16x5.72e-063.56e-03granite-3.0-1b-a400m(24L)fsdp=2, expert=4(8, 8)(64)0/0->24/1236->00.5566->0.55941.74->1.663.71->3.451.08x0.00e+002.54e-03granite-3.0-1b-a400m(24L)fsdp=2, expert=4(8, 256)(2048)0/0->24/1236->01.2134->1.25734.31->4.2111.12->9.441.18x5.72e-063.56e-03glm5.3-flash-4Lfsdp=8(ep_size=1)(8, 256)(2048)ragged_dot(0/0)272.46158.8926.311.00x(0%regression)0.00e+000.00e+00glm5.3-flash-4Lfsdp=2, expert=4(ep_size=4)(8, 8)(64)N/A(NotImplementedError) ->12/6N/A->0N/A->3.0877N/A->6.38N/A->19.62EP>1enabled)9.27e-039.59e-02glm5.3-flash-4Lfsdp=2, expert=4(ep_size=4)(8, 256)(2048)N/A(NotImplementedError) ->12/6N/A->0N/A->3.7320N/A->13.35N/A->29.14EP>1enabled)2.17e-041.84e-01m3-track5-moe-bench-bench-0-0-nbz9z, https://cloudlogging.app.goo.gl/U43TdjvmN65Zz9v29 & https://cloudlogging.app.goo.gl/cK2UjrXoNuwRGFL6A):Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.