Skip to content

feat(m3): adopt shared MoEExecutor for GraniteMoE and multi-rank EP GLM-5.3 - #6282

Open
AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-shared-moe-ep-kernelsfrom
yaoyuchen/m3-moe-glm5-granitemoe-adoption
Open

AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-shared-moe-ep-kernelsfrom
yaoyuchen/m3-moe-glm5-granitemoe-adoption

Conversation

@AusarYao28

@AusarYao28 AusarYao28 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

feat(m3): adopt shared MoEExecutor for GraniteMoE and multi-rank EP GLM-5.3

Description

Depends on #6164 (stack: #6066 -> #6152 -> #6163 -> #6164 -> this PR); review the top commit only.

Adopts the shared m3 MoEExecutor (src/maxtext/m3/core/moe.py, selecting kind="gmm_v2" for configs without use_tokamax_gmm=True or megablox=False) for GraniteMoeExperts (src/maxtext/m3/models/granitemoe/modeling_granitemoe.py) and for multi-device Expert Parallelism (ep_size > 1) on Glm5Experts (src/maxtext/m3/models/glm5_3_flash/modeling_glm5_3_flash.py), using sparse_matmul: bool = True:

  1. GraniteMoeConfig & GraniteMoeExperts (src/maxtext/m3/models/granitemoe/modeling_granitemoe.py):
    • Validates isinstance(sparse_matmul, bool) before coercion on GraniteMoeConfig.
    • Routes both the compound ("fsdp", "expert") jax.shard_map path (_ep_tp_kernel) and non-compound / single-device paths through m3_moe.grouped_matmul / MoEExecutor.fused / grouped_unsharded_fused when sparse_matmul=True (padding non-divisible ep_size > 1 decode batches with -1 indices and slicing back), preserving compound ("fsdp", "expert") expert-weight sharding (0 expert-weight all-gathers) and cutting tpu7x-8 step time by 1.08x–1.18x (1.21x–1.49x on 2-layer training sweeps).
  2. Glm5Config, Glm5Sharding, & Glm5Experts (src/maxtext/m3/configs/glm5_3_flash_config.py, src/maxtext/m3/models/glm5_3_flash/modeling_glm5_3_flash.py):
    • Adds sparse_matmul: bool = True to Glm5Config with strict isinstance(sparse_matmul, bool) validation before coercion and from_maxtext_config forwarding.
    • Enables multi-device Expert Parallelism (ep_size > 1) on Glm5MoE (previously raised NotImplementedError) by dispatching ep_size > 1 through MoEExecutor.fused (jax.lax.ragged_all_to_all + Pallas Mosaic gmm_v2 / tgmm_v2, padding non-divisible batches to ep_size with -1 indices) and folding "expert" into batch activation sharding in Glm5Sharding.from_mesh / Glm5Sharding.map_axis (3.73 GiB Peak HBM and 29.14 ms step time on fsdp=2, expert=4).
    • Preserves Glm5Experts's native jax.lax.ragged_dot path when ep_size == 1 (0% regression at EP = 1) and keeps topk_weights in jnp.float32.
  3. Shared src/maxtext/m3/core/moe.py Enhancements:
    • Adds grouped_unsharded_fused (wrapping _grouped_unsharded in a 1-device jax.shard_map over jax.local_devices()[:1] when is_tpu=True across SHARDED_EXECUTOR_KINDS) and threads backend / is_tpu into _grouped_unsharded so non-mesh callers use the resolved Pallas gmm_v2 backend without executing Pallas kernels outside jax.shard_map (the underlying gmm_v2/tgmm_v2 dispatch, (256, 128) tile alignment, and forward/TPU-bwd valid_rows masking landed in #6164; live TPU "tokamax" dispatches to tokamax.ragged_dot).

FIXES: b/580120934

Tests

1. Targeted 4-CPU Unit & Multi-Device Parity Tests (4-CPU Host Mesh)

  • tests/unit/m3/granitemoe_test.py -k "multidevice": 1 passed, 20 deselected (test_single_and_multidevice_tokamax_forward_vjp_parity_and_no_weight_all_gather on data=4, fsdp=2, expert=2, fsdp=2, tensor=2, plus non-compound EP > 1 num_local_experts=6 on fsdp=2, expert=2 and (1, 1, H) decode batch, verifying all-leaf parameter gradient parity and zero full-expert all-gather in compiled HLO with live positive control).
  • tests/unit/m3/glm5_3_flash_moe_test.py -k "multidevice": 1 passed, 10 deselected (test_multidevice_tp_and_ep_tokamax_execution_and_no_weight_all_gather on fsdp=2, tensor=2, fsdp=2, expert=2, expert=4, expert=2, tensor=2, plus (1, 1, H) decode batch, verifying all-leaf parameter gradient parity and zero full-expert all-gather in compiled HLO with live positive control).
  • tests/unit/m3/moe_test.py -k "unassigned_tail_rows": 2 passed, 34 deselected (both test_grouped_matmul_unassigned_tail_rows_zeroed_in_fwd_and_bwd_ragged_dot and test_grouped_matmul_unassigned_tail_rows_zeroed_in_fwd_and_bwd_tokamax pass).
  • Full tests/unit/m3/ (4-CPU host mesh, serial): 509 passed, 70 skipped, 46 subtests passed.

2. CPU-Hosted TPU7x AOT (tpu7x-8 and tpu7x-128) & Live tpu7x-8 (2x2x1) Benchmarks

Model (bfloat16) 8-Device tpu7x-8 Mesh (B, S) (T_step) %gmm_v2 / %tgmm_v2 (RaggedDot -> Shared MoEExecutor) Expander Conv (RaggedDot -> Shared MoEExecutor) Peak HBM (RaggedDot -> Shared MoEExecutor, GiB) Forward p50 (ms) Full Train Step p50 (ms) Train Step Speedup Loss Abs Diff Grad Rel L2 Diff
granite-3.0-1b-a400m (24L) fsdp=8 (8, 8) (64) 0/0 -> 24/12 36 -> 0 0.5566 -> 0.5594 1.74 -> 1.66 3.81 -> 3.50 1.09x 0.00e+00 2.54e-03
granite-3.0-1b-a400m (24L) fsdp=8 (8, 256) (2048) 0/0 -> 24/12 36 -> 0 1.2134 -> 1.2573 4.33 -> 4.21 11.02 -> 9.51 1.16x 5.72e-06 3.56e-03
granite-3.0-1b-a400m (24L) fsdp=2, expert=4 (8, 8) (64) 0/0 -> 24/12 36 -> 0 0.5566 -> 0.5594 1.74 -> 1.66 3.71 -> 3.45 1.08x 0.00e+00 2.54e-03
granite-3.0-1b-a400m (24L) fsdp=2, expert=4 (8, 256) (2048) 0/0 -> 24/12 36 -> 0 1.2134 -> 1.2573 4.31 -> 4.21 11.12 -> 9.44 1.18x 5.72e-06 3.56e-03
glm5.3-flash-4L fsdp=8 (ep_size=1) (8, 256) (2048) Kept on native ragged_dot (0/0) 27 2.4615 8.89 26.31 1.00x (0% regression) 0.00e+00 0.00e+00
glm5.3-flash-4L fsdp=2, expert=4 (ep_size=4) (8, 8) (64) N/A (NotImplementedError) -> 12/6 N/A -> 0 N/A -> 3.0877 N/A -> 6.38 N/A -> 19.62 New (EP>1 enabled) 9.27e-03 9.59e-02
glm5.3-flash-4L fsdp=2, expert=4 (ep_size=4) (8, 256) (2048) N/A (NotImplementedError) -> 12/6 N/A -> 0 N/A -> 3.7320 N/A -> 13.35 N/A -> 29.14 New (EP>1 enabled) 2.17e-04 1.84e-01
=== M3_TRACK5_MOE_BENCH POD_START 2026-11-11T02:03:44Z HOST=m3-track5-moe-bench-bench-0-0 ===
=== STAGE_START: hlo_and_aot_audit ===
[HLO_AUDIT] model=granite-3.0-1b-a400m mesh=fsdp8 B=8 S=8 T_step=64 M_local=64 | DEFAULT(gmm_v2=0,rdot=0,conv=36,hbm=0.5566GiB) | TOKAMAX(gmm_v2=24,tgmm_v2=12,rdot=0,conv=0,ra2a=0,hbm=0.5594GiB)
[HLO_AUDIT] model=granite-3.0-1b-a400m mesh=fsdp8 B=8 S=256 T_step=2048 M_local=2048 | DEFAULT(gmm_v2=0,rdot=0,conv=36,hbm=1.2134GiB) | TOKAMAX(gmm_v2=24,tgmm_v2=12,rdot=0,conv=0,ra2a=0,hbm=1.2573GiB)
[HLO_AUDIT] model=granite-3.0-1b-a400m mesh=fsdp2_ep4 B=8 S=8 T_step=64 M_local=64 | DEFAULT(gmm_v2=0,rdot=0,conv=36,hbm=0.5566GiB) | TOKAMAX(gmm_v2=24,tgmm_v2=12,rdot=0,conv=0,ra2a=0,hbm=0.5594GiB)
[HLO_AUDIT] model=granite-3.0-1b-a400m mesh=fsdp2_ep4 B=8 S=256 T_step=2048 M_local=2048 | DEFAULT(gmm_v2=0,rdot=0,conv=36,hbm=1.2134GiB) | TOKAMAX(gmm_v2=24,tgmm_v2=12,rdot=0,conv=0,ra2a=0,hbm=1.2573GiB)
[HLO_AUDIT] model=glm5.3-flash-4L mesh=fsdp2_ep4 B=8 S=8 T_step=64 M_local=64 | DEFAULT(gmm_v2=0,rdot=0,conv=27,hbm=1.6861GiB) | TOKAMAX(gmm_v2=12,tgmm_v2=6,rdot=0,conv=0,ra2a=60,hbm=3.0877GiB)
[HLO_AUDIT] model=glm5.3-flash-4L mesh=fsdp2_ep4 B=8 S=256 T_step=2048 M_local=2048 | DEFAULT(gmm_v2=0,rdot=0,conv=27,hbm=2.4615GiB) | TOKAMAX(gmm_v2=12,tgmm_v2=6,rdot=0,conv=0,ra2a=60,hbm=3.732GiB)
=== STAGE_END: hlo_and_aot_audit ===
=== STAGE_START: step_latency_benchmark ===
[LATENCY_BENCH] model=granite-3.0-1b-a400m mesh=fsdp8 B=8 S=8 T_step=64 M_local=64 | fwd_def_p50=1.74ms fwd_tok_p50=1.66ms fwd_speedup=1.05x | step_def_p50=3.81ms step_tok_p50=3.50ms step_speedup=1.09x
[LATENCY_BENCH] model=granite-3.0-1b-a400m mesh=fsdp8 B=8 S=256 T_step=2048 M_local=2048 | fwd_def_p50=4.33ms fwd_tok_p50=4.21ms fwd_speedup=1.03x | step_def_p50=11.02ms step_tok_p50=9.51ms step_speedup=1.16x
[LATENCY_BENCH] model=granite-3.0-1b-a400m mesh=fsdp2_ep4 B=8 S=8 T_step=64 M_local=64 | fwd_def_p50=1.74ms fwd_tok_p50=1.66ms fwd_speedup=1.05x | step_def_p50=3.71ms step_tok_p50=3.45ms step_speedup=1.08x
[LATENCY_BENCH] model=granite-3.0-1b-a400m mesh=fsdp2_ep4 B=8 S=256 T_step=2048 M_local=2048 | fwd_def_p50=4.31ms fwd_tok_p50=4.21ms fwd_speedup=1.02x | step_def_p50=11.12ms step_tok_p50=9.44ms step_speedup=1.18x
[LATENCY_BENCH] model=glm5.3-flash-4L mesh=fsdp2_ep4 B=8 S=8 T_step=64 M_local=64 | fwd_def_p50=3.39ms fwd_tok_p50=6.38ms fwd_speedup=0.53x | step_def_p50=9.84ms step_tok_p50=19.62ms step_speedup=0.50x
[LATENCY_BENCH] model=glm5.3-flash-4L mesh=fsdp2_ep4 B=8 S=256 T_step=2048 M_local=2048 | fwd_def_p50=8.73ms fwd_tok_p50=13.35ms fwd_speedup=0.65x | step_def_p50=25.41ms step_tok_p50=29.14ms step_speedup=0.87x
=== STAGE_END: step_latency_benchmark ===
=== M3_TRACK5_MOE_BENCH POD_DONE 2026-11-11T02:21:06Z ===
M3_TRACK5_MOE_BENCH_EXIT_CODE=0

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adopts a unified MoEExecutor for both GraniteMoE and GLM-5.3 models within the m3 framework. By centralizing MoE execution logic, the changes enable multi-device Expert Parallelism for GLM-5.3 and optimize performance for GraniteMoE through improved kernel dispatch and better utilization of TPU hardware. The implementation includes robust configuration validation, improved sharding logic, and comprehensive unit testing to ensure numerical parity and performance gains.

Highlights

  • MoEExecutor Adoption: Integrated the shared m3 MoEExecutor into GraniteMoeExperts and Glm5Experts, enabling multi-device Expert Parallelism (EP) for GLM-5.3 and improving performance for GraniteMoE.
  • Performance Improvements: Achieved significant speedups (1.09x–1.18x) on TPU7x hardware by routing operations through grouped_matmul and leveraging Pallas Mosaic kernels (gmm_v2/tgmm_v2).
  • Expert Parallelism Support: Enabled multi-device EP (ep_size > 1) for Glm5MoE by dispatching through MoEExecutor.fused and folding expert sharding into batch activation sharding.
  • Core Enhancements: Improved m3 core MoE logic, including direct lowering to Pallas kernels, tail-row masking to prevent NaNs, and better handling of empty meshes.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-moe-glm5-granitemoe-adoption branch from f93f3f9 to ee2aff7 Compare November 12, 2026 09:54
@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.81818% with 62 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...axtext/m3/models/granitemoe/modeling_granitemoe.py 64.63% 21 Missing and 8 partials ⚠️
...xt/m3/models/glm5_3_flash/modeling_glm5_3_flash.py 75.00% 18 Missing and 10 partials ⚠️
src/maxtext/m3/core/moe.py 66.66% 3 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

@AusarYao28 AusarYao28 removed skip-ci-tests Add this label to the PR to skip running CI tests gemini-review labels Nov 12, 2026
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-moe-glm5-granitemoe-adoption branch from ee2aff7 to 0beb5c1 Compare November 12, 2026 22:22
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec partially met; TPU tail-row masking incomplete and multi-host device binding bug.

Top issues:

  1. ⚠️ [Description] PR description omits the change to _validate_generic_maxtext_kwargs in modeling_glm5_3_flash.py, which adds call-stack frame inspection for loss_fn. → Document why _validate_generic_maxtext_kwargs was modified, or preferably handle trainer kwargs explicitly without reflection.
  2. 🔴 [Bug] Unassigned tail rows in _pallas_grouped_matmul and _pallas_grouped_matmul_bwd are masked on CPU, but lhs remains unmasked on TPU before padding for gmm_v2 and tgmm_v2. If unassigned tail rows contain NaNs, the contraction in tgmm_v2 will propagate NaN into weight gradient drhs; see inline comments on src/maxtext/m3/core/moe.py. → Mask lhs with valid_rows before padding on TPU in both forward and backward passes.
  3. [Alignment] _validate_generic_maxtext_kwargs uses inspect.currentframe() to dynamically walk caller frames and detect loss_fn in maxtext/trainers. This reflection is fragile in JAX tracing and inconsistent with sibling M3 models; see inline comment on modeling_glm5_3_flash.py. → Handle or ignore generic trainer kwargs (enable_dropout, decoder_target_tokens) directly at the model interface without inspecting frames.
  4. [Bug] grouped_unsharded_fused builds a 1-device mesh using jax.devices()[:1], which crashes on multi-host TPU slices because worker hosts > 0 do not own device 0; see inline comment on src/maxtext/m3/core/moe.py. → Use jax.local_devices()[:1].
  5. [Alignment] GraniteMoeExperts branches on backend in ("tokamax", "megablox") and forwards backend to m3_moe.grouped_matmul, which rejects "megablox" with ValueError; see inline comment on modeling_granitemoe.py. → Standardize on backend == "tokamax".
Inferred spec & validation
  • Sources: PR description; linked Buganizer ticket b/580120934 (not readable here); code before/after.
  • Problem: GraniteMoe and GLM-5.3-Flash lacked unified MoEExecutor adoption, leaving GLM-5.3-Flash without multi-device EP (ep_size > 1) support and exposing potential tail-row padding and alignment hazards on grouped matmuls.
Spec item Implementation Validating test
R1 (stated): GraniteMoeConfig and Glm5Config support sparse_matmul: bool = True with strict boolean validation ✅ met glm5_3_flash_config_test.py::test_sparse_matmul_validation_and_sharding_axes ✅, granitemoe_test.py::test_sparse_matmul_config_validation_and_sharding_axes ✅
R2 (stated): Enable multi-device EP (ep_size > 1) on Glm5MoE via MoEExecutor.fused without expert-weight all-gathers ✅ met glm5_3_flash_moe_test.py::test_multidevice_tp_and_ep_tokamax_execution_and_no_weight_all_gather ✅
R3 (stated): Preserve Glm5Experts native ragged_dot execution path when ep_size == 1 ✅ met glm5_3_flash_moe_test.py::test_single_device_and_small_token_tokamax_forward_and_vjp_parity ✅
R4 (stated): Zero-mask unassigned tail rows (row_idx >= sum(group_sizes)) across ragged_dot and tokamax forward and backward ⚠️ partial moe_test.py::test_grouped_matmul_unassigned_tail_rows_zeroed_in_fwd_and_bwd ⚠️ superficial (tested with is_tpu=False, masking unmasked TPU lhs_pad)
R5 (stated): Wrap grouped_unsharded_fused in a 1-device shard_map on TPU when no mesh is in scope ⚠️ partial moe_test.py::test_grouped_unsharded_fused_without_active_mesh ⚠️ superficial (tested only on CPU; multi-host TPU fails on jax.devices()[:1])
C1 (stated): Backward compatibility and no performance regression for ep_size == 1 ✅ preserved glm5_3_flash_moe_test.py::test_ep_sharding_guard ✅
C2 (stated): Compound ("fsdp", "expert") weight sharding in GraniteMoe preserved with 0 expert all-gathers ✅ preserved granitemoe_test.py::test_single_and_multidevice_tokamax_forward_vjp_parity_and_no_weight_all_gather ✅
C3 (inferred): Standard trainer interface kwargs handled without runtime stack inspection ❌ violated ❌ none

Comment thread src/maxtext/m3/core/moe.py Outdated
if not is_tpu:
return jax.lax.ragged_dot(
lhs,
lhs_safe = jnp.where(valid_rows, lhs, jnp.zeros((), dtype=lhs.dtype))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 lhs_safe is computed inside if not is_tpu: and only used for the CPU ragged_dot fallback. On TPU (line 187), lhs_pad = jnp.pad(lhs, ...) pads the unmasked lhs, meaning unassigned tail rows (row_idx >= valid_count) containing garbage or NaNs are passed directly into gmm_v2. Compute lhs_safe = jnp.where(valid_rows, lhs, jnp.zeros((), dtype=lhs.dtype)) before if not is_tpu: and pad lhs_safe on TPU.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 752e2bf4af (and parent #6164 53c481bad3): hoisted lhs_safe = jnp.where(valid_rows, lhs, 0) above if not is_tpu: in _pallas_grouped_matmul and grouped_matmul so TPU gmm_v2 / tokamax.ragged_dot paths zero unassigned tail rows before padding.


if backend_kind == "tokamax":
if not is_tpu:
lhs_safe = jnp.where(valid_rows, lhs, jnp.zeros((), dtype=lhs.dtype))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 lhs_safe in the backward pass is only defined within if not is_tpu:. On TPU (line 252), lhs_pad = jnp.pad(lhs, ...) passes the unmasked lhs to tgmm_v2.tgmm_v2(lhs=lhs_pad, rhs=grad_pad). If unassigned tail rows in lhs contain NaNs, the contraction with grad_pad will propagate NaN into weight gradients (drhs). Define lhs_safe outside if not is_tpu: and pass lhs_safe to lhs_pad.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 752e2bf4af (and parent #6164 53c481bad3): hoisted lhs_safe = jnp.where(valid_rows, lhs, 0) above if not is_tpu: in _pallas_grouped_matmul_bwd and passed lhs_safe into lhs_pad = jnp.pad(lhs_safe, ...) before tgmm_v2.tgmm_v2 on TPU.

Comment thread src/maxtext/m3/core/moe.py Outdated
projections = (gate_up_proj.astype(x.dtype), down_proj.astype(x.dtype))
backend = executor.kind if executor.kind in ("ragged_dot", "tokamax") else "ragged_dot"
if is_tpu and backend == "tokamax":
mesh_1d = jax.sharding.Mesh(jax.devices()[:1], ("_moe_1d",))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 On multi-host TPU slices (e.g. v4-16, v5p-128), jax.devices()[:1] selects device 0 on worker host 0. Any non-zero worker host will crash with a device locality error because device 0 is not owned by that process. Use jax.local_devices()[:1] so each host binds to its local TPU device.

Suggested change
mesh_1d = jax.sharding.Mesh(jax.devices()[:1], ("_moe_1d",))
mesh_1d = jax.sharding.Mesh(jax.local_devices()[:1], ("_moe_1d",))

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 752e2bf4af: grouped_unsharded_fused now builds its single-device mesh from jax.local_devices()[:1] rather than jax.devices()[:1] so multi-host TPU processes only reference a host-local addressable device.

precision=precision,
)
dn = down_w.astype(dt)
if backend in ("tokamax", "megablox"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 m3_moe.grouped_matmul only accepts backend="ragged_dot" or backend="tokamax", raising a ValueError if passed "megablox". In addition, MoEExecutor.kind is typed as Literal["dense", "ragged_dot", "tokamax"]. Standardize on backend == "tokamax" here and in lines 774, 784, 799, and 803.

Suggested change
if backend in ("tokamax", "megablox"):
if backend == "tokamax":

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 752e2bf4af (with SHARDED_EXECUTOR_KINDS = ('gmm_v2', 'tokamax') in #6164 53c481bad3): GraniteMoeExperts and Glm5MoE now resolve executor kinds via moe_from_config ('gmm_v2' for megablox=True, 'tokamax' for use_tokamax_gmm=True) and check backend in m3_moe.SHARDED_EXECUTOR_KINDS everywhere instead of passing 'megablox' to grouped_matmul.

Comment on lines +2036 to +2045
import inspect

is_generic_trainer = False
frame = inspect.currentframe()
while frame is not None:
if frame.f_code.co_name == "loss_fn" and "maxtext/trainers" in frame.f_code.co_filename:
is_generic_trainer = True
break
frame = frame.f_back

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Inspecting the runtime call stack via inspect.currentframe() to check for loss_fn in maxtext/trainers is fragile under JIT tracing, function decorators, or custom trainer/evaluation loops. Consistent with sibling M3 models (such as GraniteMoe and Qwen3), handle or ignore generic trainer kwargs (enable_dropout, decoder_target_tokens) directly at the layer interface instead of inspecting stack frames.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 752e2bf4af: removed the inspect.currentframe() stack inspection and restored Glm5Model._validate_generic_maxtext_kwargs to its fail-closed origin/main behavior (verified by test_generic_maxtext_kwargs_validation).

Comment thread tests/unit/m3/moe_test.py Outdated
ref_dlhs, ref_drhs = ref_vjp(cot_valid)

out, vjp_fn = jax.vjp(
lambda l, r: m3_moe.grouped_matmul(l, r, group_sizes, backend=backend, is_tpu=False),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This test specifies is_tpu=False for both parameterized backends, which routes "tokamax" through the CPU ragged_dot fallback and leaves the TPU Pallas kernels (gmm_v2 / tgmm_v2) and padding logic unexercised. Add a TPU test (or test double) to verify that unassigned tail rows are zeroed in lhs_pad on TPU.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 752e2bf4af: added a mocked-TPU (is_tpu=True) test injecting NaNs into unassigned tail rows [sum(group_sizes):M] and verifying gmm_v2.gmm_v2 and tgmm_v2.tgmm_v2 receive zero-masked lhs_pad/grad_pad with finite zeroed tail outputs and gradients (plus live 4-chip TPU v5p hardware execution).

@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-shared-moe-ep-kernels branch from 2f414d9 to 53c481b Compare November 13, 2026 09:44
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-moe-glm5-granitemoe-adoption branch from 0beb5c1 to 752e2bf Compare November 13, 2026 09:50
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-shared-moe-ep-kernels branch from 53c481b to 91ef25a Compare November 13, 2026 10:09
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-moe-glm5-granitemoe-adoption branch from 752e2bf to 3bd58c0 Compare November 13, 2026 10:15
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-shared-moe-ep-kernels branch from 91ef25a to 70f09c5 Compare November 13, 2026 20:05
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-moe-glm5-granitemoe-adoption branch from 3bd58c0 to 15dc9bb Compare November 13, 2026 20:10
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-shared-moe-ep-kernels branch from 70f09c5 to e60cc45 Compare November 14, 2026 00:07
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-moe-glm5-granitemoe-adoption branch from 15dc9bb to 789e1d8 Compare November 14, 2026 00:12
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

…LM-5.3

- Route GraniteMoeExperts compound ('fsdp', 'expert') and non-compound paths
  through shared m3_moe.grouped_matmul and MoEExecutor.fused when
  sparse_matmul=True, preserving zero expert-weight all-gather.
- Route Glm5Experts and Glm5MoE through shared MoEExecutor.fused when
  ep_size > 1 and sparse_matmul=True (keeping routing weights in float32 and
  sharding batch activations across ('data', 'fsdp', 'expert')), while keeping
  ep_size == 1 on native ragged_dot.
- Mask unassigned tail rows [sum(group_sizes):M] in forward and backward across
  m3_moe.grouped_matmul backends and add grouped_unsharded_fused fallback when
  no Mesh is in scope.

FIXES: b/580120934
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-shared-moe-ep-kernels branch from e60cc45 to 3d25e26 Compare November 17, 2026 03:06
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-moe-glm5-granitemoe-adoption branch from 789e1d8 to 4d7dd82 Compare November 17, 2026 03:11
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant