Skip to content

fix(vllm): support GPT-OSS on the fused MoE path of MaxTextForCausalLM - #6028

Open
chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-gpt-oss-vllm-decode-fused-moe
Open

chiajunglien wants to merge 1 commit into
mainfrom
emma/fix-gpt-oss-vllm-decode-fused-moe

Conversation

@chiajunglien

@chiajunglien chiajunglien commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Description

vllm_decode of GPT-OSS-20B with MaxTextForCausalLM crashed in tpu-inference's fused MoE kernel (GMM_v2), used by RoutedMoE.fused_moe_matmul with vLLM attention, which GPT-OSS did not support:

  1. The vLLM adapter pads the MoE MLP dim for GMM_v2 but only read moe_intermediate_size from the HF config. GPT-OSS only has intermediate_size (2880), so nothing was padded: size_n=720 should be divisible by 2 * num_lanes when fuse_act is enabled.
  2. The kernel got mlp_activations[0] ("sigmoid") instead of GPT-OSS's clamped SwiGLU, the kernel's "swigluoai".
  3. The expert biases were dropped (w1_bias=None, w2_bias=None).
  4. Under vLLM's logical axis rules, the biases' ("exp", "activation_mlp") sharding put one mesh axis on two dimensions, which tpu-inference rejects when it creates the model under its mesh.

The other path in the bug, vLLM's own GPT-OSS (no vllm_hf_overrides), ignores load_parameters_path and loads the full HF checkpoint through host memory, hence the host OOM.

Changes

  • get_moe_intermediate_size(): falls back to intermediate_size when every MLP is MoE; the adapter uses it (GPT-OSS: 2880 → 3072 at TP ≤ 4, 4096 at TP 8).
  • fused_moe_activation(): GPT-OSS → "swigluoai"; raises if the settings differ from the kernel's hard-coded alpha/limit.
  • fuse_moe_biases_for_kernel(): lays out the gate/up biases like the fused wi (per-shard padding; per-shard [gate | up] for the TP kernel, [gate | up] for the EP kernel). fused_moe_matmul passes the biases to the kernel.
  • Under vLLM attention, the biases use activation_mlp_moe, sharded like the expert kernels.
  • vllm_decode warns when load_parameters_path is set without MaxTextForCausalLM. It reuses uses_maxtext_vllm_adapter, moved to convert_utils so that vllm_decode does not import the RL rollout module (which patches Tunix at import time).
  • test_gpt_oss_rl.sh and test_gpt_oss_sft.sh run vllm_decode again on the converted checkpoint (before training) and on the checkpoint the run produces (after training): MaxText model, TP 4 x DP 8, prefuse_moe_weights=True, greedy. The training steps are unchanged.

Notes

BUGS: b/544614464

Tests

Unit tests

CPU, 4 virtual devices: fused_moe_qwix_test and convert_utils_test: 42 passed, 2 skipped (TPU only). The GPT-OSS tests live in fused_moe_qwix_test next to the other fused_moe_matmul tests and share its _fake_tpu_inference.

  • GPT-OSS MoE through fused_moe_matmul (real loader, CPU emulation of the kernel) matches the dense implementation: unfused, prefused, TP 2 / TP 4, TP 2 x EP 2. GptOssFusedMoeTPUTest does the same with the real kernel at smaller dims (emb 256, MLP 360, 16 experts); its logic was checked on CPU with the emulated kernel.
  • Zeroed biases or a wrong bias layout make it fail.
  • A small GPT-OSS model builds under vLLM's rules and mesh with biases sharded like the expert kernels (DuplicateSpecError without the fix).

E2E on TPU v6e

v6e-32, Pathways (the maxtext_e2e_tpu_post_training setup), image tpu_post_training:047d606 + this PR, test_gpt_oss_rl.sh, checkpoint e2e-40609490996. Run 3 also has #6027.

Step Run 2: this PR Run 3: + #6027
1. decode converted ckpt ✅ analysisThe user wants a suggestion of some famous landmarks in London. ... ✅ analysisThe user asks: "Suggest some famous landmarks in London." ...
2. RL (GSM8K) ✅ 81.25% → 95.0% ✅ 81.25% → 95.0%
3. decode scanned RL ckpt ❌ duplication suggests we used when factor ... ✅ analysisUser asks: "Suggest some famous landmarks in London." ...

Logs: run 2 (Step 1 output at line 2207, Step 3 at 6143), run 3 (2219, 6155). Run 2 sampled at temperature 1.0. Both runs also applied #5988 (no effect with TP only). The SFT script's decode steps are the same commands as the RL script's but have not been run end to end yet. Not validated on v5p.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This PR enables support for GPT-OSS-20B models on the fused MoE path within MaxTextForCausalLM by addressing compatibility issues with the tpu-inference GMM_v2 kernel. Key fixes include correct MLP dimension calculation, proper activation mapping, and handling of expert biases and their sharding. Additionally, the PR adds robust unit testing and updates the E2E test script to validate the RL pipeline.

Highlights

  • MoE Intermediate Size: Added get_moe_intermediate_size to correctly identify the MLP dimension for models like GPT-OSS where all MLPs are MoE.
  • Fused Kernel Activation: Added fused_moe_activation to map GPT-OSS's clamped SwiGLU to the kernel's swigluoai activation.
  • Bias Handling: Implemented fuse_moe_biases_for_kernel to correctly lay out and pad expert biases for the fused MoE kernel.
  • Sharding Fix: Updated bias sharding rules to use activation_mlp_moe for vLLM compatibility, preventing DuplicateSpecError.
  • Inference Warnings: Added a warning in vllm_decode when load_parameters_path is used without MaxTextForCausalLM to prevent host OOM.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met; the fix is well-tested and addresses the core issues. Minor robustness nit found.

Top issues:

  1. 🟠 [Bug] Accessing self.mesh.shape[axis] can raise a KeyError if the mapped axis is not in the mesh. → Use self.mesh.shape.get(axis, 1) (see inline comment on moe.py).
Inferred spec & validation
  • Sources: PR description; code before/after.
  • Problem: vllm_decode of GPT-OSS-20B crashed in the fused MoE kernel due to missing padding, incorrect activation, dropped biases, and incompatible sharding rules.
Spec item Implementation Validating test
R1 (stated): Read and pad GPT-OSS's MoE MLP dim for the GMM_v2 kernel ✅ fused_moe_gpt_oss_test.py::test_matches_dense ✅
R2 (stated): Use the kernel's "swigluoai" activation for GPT-OSS ✅ fused_moe_gpt_oss_test.py::test_matches_dense ✅
R3 (stated): Lay out gate and up biases in the format expected by the kernel ✅ fused_moe_gpt_oss_test.py::test_matches_dense ✅
R4 (stated): Provide valid vLLM sharding rules for the biases ✅ fused_moe_gpt_oss_test.py::test_abstract_model_has_valid_shardings ✅

Comment thread src/maxtext/layers/moe.py Resolved
@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/moe.py 91.66% 1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@chiajunglien
chiajunglien force-pushed the emma/fix-gpt-oss-vllm-decode-fused-moe branch from a48c8bf to 3fba404 Compare October 29, 2026 06:20
@chiajunglien
chiajunglien force-pushed the emma/fix-gpt-oss-vllm-decode-fused-moe branch 2 times, most recently from cf589ca to 3ba1d13 Compare November 2, 2026 06:26
@chiajunglien
chiajunglien force-pushed the emma/fix-gpt-oss-vllm-decode-fused-moe branch from 3ba1d13 to 2b67d37 Compare November 4, 2026 06:49
Comment thread tests/unit/fused_moe_gpt_oss_test.py Resolved
@chiajunglien
chiajunglien force-pushed the emma/fix-gpt-oss-vllm-decode-fused-moe branch from 2b67d37 to a217009 Compare November 9, 2026 03:13
@chiajunglien chiajunglien removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 9, 2026
vllm_decode of GPT-OSS with MaxTextForCausalLM crashed in tpu-inference's
fused MoE kernel (GMM_v2), which RoutedMoE uses with vLLM attention:

- The vLLM adapter pads the MoE MLP dim for GMM_v2 but only read
  moe_intermediate_size from the HF config. GPT-OSS only has
  intermediate_size (2880), so it was not padded and the kernel failed
  with "size_n=720 should be divisible by 2 * num_lanes".
- The kernel got mlp_activations[0] ("sigmoid") instead of GPT-OSS's
  clamped SwiGLU, which the kernel calls "swigluoai".
- The expert biases were dropped (w1_bias/w2_bias=None). Under vLLM's
  logical axis rules, their ("exp", "activation_mlp") sharding also put
  one mesh axis on two dimensions, which fails when tpu-inference creates
  the model under its mesh.

Use intermediate_size when every MLP is MoE and moe_intermediate_size is
missing; map GPT-OSS to "swigluoai" (raising for settings that differ
from the kernel's hard-coded ones); pass the expert biases in the
kernel's fused layout (per-shard padding, per-shard [gate | up] for the
TP kernel, [gate | up] for the EP kernel); and shard the biases like the
expert kernels under vLLM.

vllm_decode now warns when load_parameters_path is set without
MaxTextForCausalLM: vLLM's own implementation then ignores the
checkpoint and loads the full HF checkpoint through host memory, which
is the host OOM in the bug. It reuses uses_maxtext_vllm_adapter, which
moves to convert_utils so that vllm_decode does not import the RL
rollout module (that module patches Tunix at import time).
test_gpt_oss_rl.sh and test_gpt_oss_sft.sh run vllm_decode again, on the
converted checkpoint and on the checkpoint the run produces, with greedy
sampling.

Bug: b/544614464
@chiajunglien
chiajunglien force-pushed the emma/fix-gpt-oss-vllm-decode-fused-moe branch from a217009 to 0393948 Compare November 10, 2026 03:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants