Skip to content

feat(m3): support forced_routed_experts router replay in m3/core/moe.py - #6408

Open
AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-moe-dropless-overflow-bufferfrom
yaoyuchen/m3-moe-router-replay
Open

AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-moe-dropless-overflow-bufferfrom
yaoyuchen/m3-moe-router-replay

Conversation

@AusarYao28

Copy link
Copy Markdown
Collaborator

Description

Depends on #6281 (stacked on yaoyuchen/m3-moe-dropless-overflow-buffer; review the top commit only).

Adds forced_routed_experts router replay support to src/maxtext/m3/core/moe.py for deterministic MoE debugging, distillation, and RL/on-policy replay without altering default training HLO:

  • Threads optional forced_routed_experts (shape [..., num_experts_per_tok], UNSET_ROUTED_EXPERT = -1 for padded/inactive slots) through route_top_k, apply_forced_routed_experts, MoEExecutor, routed_experts, routed_experts_fused, and grouped_unsharded_fused.
  • Preserves byte-identical HLO when forced_routed_experts is None.
  • Fails closed on invalid shapes, non-integer dtypes, or out-of-range expert IDs outside [-1, num_experts), while clamping -1 slots before take_along_axis and zeroing inactive slot weights and denominators so -1 slots contribute exact 0.0 to outputs, group_sizes, and gradients.
  • Partitions forced_routed_experts along (batch, length) inside shard_map across both EP == 1 and EP > 1 (ragged_all_to_all + dropless overflow buffer).

Tests

  • CPU unit tests (4-device host mesh, serial):
    • tests/unit/m3/moe_router_replay_test.py + tests/unit/m3/moe_test.py -k "router_replay or dropless": 40 passed (30 router-replay tests + 10 dropless tests; covers zero-overhead HLO parity, exact 0.0 forward/backward equivalence on natural top-k indices, non-natural forced overrides, -1 partial and all--1 row padding, and fail-closed validation).
  • Pre-commit linters & audits: pyink, pylint (10.00/10), codespell, and m3 contract audits clean.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

- Thread optional forced_routed_experts (shape [..., num_experts_per_tok],
  UNSET_ROUTED_EXPERT = -1 for padded/inactive slots) through route_top_k,
  apply_forced_routed_experts, MoEExecutor, routed_experts,
  routed_experts_fused, and grouped_unsharded_fused.
- Preserve 100% identical HLO when forced_routed_experts is None.
- Validate shape, integer dtype, and concrete ID range [-1, num_experts)
  failing closed on invalid shapes/dtypes/IDs, while clamping -1 slots
  before take_along_axis and zeroing inactive slot weights and denominators
  so -1 slots contribute exact 0.0 to outputs, group_sizes, and gradients.
- Partition forced_routed_experts along (batch, length) inside shard_map
  across EP == 1 and EP > 1 (ragged_all_to_all + dropless overflow buffer).
- Add 30 4-CPU unit tests in tests/unit/m3/moe_router_replay_test.py
  covering zero-overhead HLO parity, exact 0.0 fwd+bwd equivalence on
  natural top_k indices, non-natural forced overrides, -1 partial and
  all--1 row padding, and fail-closed validation.
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a mechanism for forced expert routing in Mixture-of-Experts (MoE) models. By allowing users to provide explicit expert indices, it facilitates advanced debugging and reinforcement learning workflows while maintaining full compatibility with existing training pipelines. The implementation ensures that forced routing remains deterministic and preserves performance characteristics, including zero-overhead parity when the feature is unused.

Highlights

  • Router Replay Support: Added support for forced_routed_experts in m3/core/moe.py to enable deterministic MoE debugging, distillation, and RL/on-policy replay without impacting default training HLO.
  • Robust Validation: Implemented strict validation for forced_routed_experts shapes, dtypes, and expert ID ranges, with fail-closed behavior for invalid inputs.
  • Padding Handling: Introduced UNSET_ROUTED_EXPERT = -1 to mark inactive slots, ensuring they contribute exactly 0.0 to outputs and gradients.
  • Testing: Added comprehensive unit tests in tests/unit/m3/moe_router_replay_test.py covering HLO parity, forward/backward equivalence, and validation edge cases.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@AusarYao28 AusarYao28 removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 44.15584% with 129 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/m3/core/moe.py 44.15% 101 Missing and 28 partials ⚠️

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant