Repository navigation
fix(vllm): support GPT-OSS on the fused MoE path of MaxTextForCausalLM - #6028
chiajunglien wants to merge 1 commit into
Conversation
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This PR enables support for GPT-OSS-20B models on the fused MoE path within MaxTextForCausalLM by addressing compatibility issues with the tpu-inference GMM_v2 kernel. Key fixes include correct MLP dimension calculation, proper activation mapping, and handling of expert biases and their sharding. Additionally, the PR adds robust unit testing and updates the E2E test script to validate the RL pipeline. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met; the fix is well-tested and addresses the core issues. Minor robustness nit found.
Top issues:
- 🟠 [Bug] Accessing
self.mesh.shape[axis]can raise aKeyErrorif the mapped axis is not in the mesh. → Useself.mesh.shape.get(axis, 1)(see inline comment onmoe.py).
Inferred spec & validation
- Sources: PR description; code before/after.
- Problem:
vllm_decodeof GPT-OSS-20B crashed in the fused MoE kernel due to missing padding, incorrect activation, dropped biases, and incompatible sharding rules.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Read and pad GPT-OSS's MoE MLP dim for the GMM_v2 kernel | ✅ | fused_moe_gpt_oss_test.py::test_matches_dense ✅ |
| R2 (stated): Use the kernel's "swigluoai" activation for GPT-OSS | ✅ | fused_moe_gpt_oss_test.py::test_matches_dense ✅ |
| R3 (stated): Lay out gate and up biases in the format expected by the kernel | ✅ | fused_moe_gpt_oss_test.py::test_matches_dense ✅ |
| R4 (stated): Provide valid vLLM sharding rules for the biases | ✅ | fused_moe_gpt_oss_test.py::test_abstract_model_has_valid_shardings ✅ |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
a48c8bf to
3fba404
Compare
cf589ca to
3ba1d13
Compare
3ba1d13 to
2b67d37
Compare
2b67d37 to
a217009
Compare
vllm_decode of GPT-OSS with MaxTextForCausalLM crashed in tpu-inference's
fused MoE kernel (GMM_v2), which RoutedMoE uses with vLLM attention:
- The vLLM adapter pads the MoE MLP dim for GMM_v2 but only read
moe_intermediate_size from the HF config. GPT-OSS only has
intermediate_size (2880), so it was not padded and the kernel failed
with "size_n=720 should be divisible by 2 * num_lanes".
- The kernel got mlp_activations[0] ("sigmoid") instead of GPT-OSS's
clamped SwiGLU, which the kernel calls "swigluoai".
- The expert biases were dropped (w1_bias/w2_bias=None). Under vLLM's
logical axis rules, their ("exp", "activation_mlp") sharding also put
one mesh axis on two dimensions, which fails when tpu-inference creates
the model under its mesh.
Use intermediate_size when every MLP is MoE and moe_intermediate_size is
missing; map GPT-OSS to "swigluoai" (raising for settings that differ
from the kernel's hard-coded ones); pass the expert biases in the
kernel's fused layout (per-shard padding, per-shard [gate | up] for the
TP kernel, [gate | up] for the EP kernel); and shard the biases like the
expert kernels under vLLM.
vllm_decode now warns when load_parameters_path is set without
MaxTextForCausalLM: vLLM's own implementation then ignores the
checkpoint and loads the full HF checkpoint through host memory, which
is the host OOM in the bug. It reuses uses_maxtext_vllm_adapter, which
moves to convert_utils so that vllm_decode does not import the RL
rollout module (that module patches Tunix at import time).
test_gpt_oss_rl.sh and test_gpt_oss_sft.sh run vllm_decode again, on the
converted checkpoint and on the checkpoint the run produces, with greedy
sampling.
Bug: b/544614464
a217009 to
0393948
Compare
Description
vllm_decodeof GPT-OSS-20B withMaxTextForCausalLMcrashed in tpu-inference's fused MoE kernel (GMM_v2), used byRoutedMoE.fused_moe_matmulwith vLLM attention, which GPT-OSS did not support:moe_intermediate_sizefrom the HF config. GPT-OSS only hasintermediate_size(2880), so nothing was padded:size_n=720 should be divisible by 2 * num_lanes when fuse_act is enabled.mlp_activations[0]("sigmoid") instead of GPT-OSS's clamped SwiGLU, the kernel's"swigluoai".w1_bias=None, w2_bias=None).("exp", "activation_mlp")sharding put one mesh axis on two dimensions, which tpu-inference rejects when it creates the model under its mesh.The other path in the bug, vLLM's own GPT-OSS (no
vllm_hf_overrides), ignoresload_parameters_pathand loads the full HF checkpoint through host memory, hence the host OOM.Changes
get_moe_intermediate_size(): falls back tointermediate_sizewhen every MLP is MoE; the adapter uses it (GPT-OSS: 2880 → 3072 at TP ≤ 4, 4096 at TP 8).fused_moe_activation(): GPT-OSS →"swigluoai"; raises if the settings differ from the kernel's hard-coded alpha/limit.fuse_moe_biases_for_kernel(): lays out the gate/up biases like the fusedwi(per-shard padding; per-shard[gate | up]for the TP kernel,[gate | up]for the EP kernel).fused_moe_matmulpasses the biases to the kernel.activation_mlp_moe, sharded like the expert kernels.vllm_decodewarns whenload_parameters_pathis set withoutMaxTextForCausalLM. It reusesuses_maxtext_vllm_adapter, moved toconvert_utilsso thatvllm_decodedoes not import the RL rollout module (which patches Tunix at import time).test_gpt_oss_rl.shandtest_gpt_oss_sft.shrunvllm_decodeagain on the converted checkpoint (before training) and on the checkpoint the run produces (after training): MaxText model, TP 4 x DP 8,prefuse_moe_weights=True, greedy. The training steps are unchanged.Notes
prefuse_moe_weights=True(unfused weights do not give the per-shard layout the TP kernel needs; not new). TP x EP with prefused weights is fixed by fix(vllm): use [gate | up] prefused MoE layout under expert parallelism #5988.BUGS: b/544614464
Tests
Unit tests
CPU, 4 virtual devices:
fused_moe_qwix_testandconvert_utils_test: 42 passed, 2 skipped (TPU only). The GPT-OSS tests live infused_moe_qwix_testnext to the otherfused_moe_matmultests and share its_fake_tpu_inference.fused_moe_matmul(real loader, CPU emulation of the kernel) matches the dense implementation: unfused, prefused, TP 2 / TP 4, TP 2 x EP 2.GptOssFusedMoeTPUTestdoes the same with the real kernel at smaller dims (emb 256, MLP 360, 16 experts); its logic was checked on CPU with the emulated kernel.DuplicateSpecErrorwithout the fix).E2E on TPU v6e
v6e-32, Pathways (the
maxtext_e2e_tpu_post_trainingsetup), imagetpu_post_training:047d606+ this PR,test_gpt_oss_rl.sh, checkpointe2e-40609490996. Run 3 also has #6027.analysisThe user wants a suggestion of some famous landmarks in London. ...analysisThe user asks: "Suggest some famous landmarks in London." ...duplication suggests we used when factor ...analysisUser asks: "Suggest some famous landmarks in London." ...Logs: run 2 (Step 1 output at line 2207, Step 3 at 6143), run 3 (2219, 6155). Run 2 sampled at temperature 1.0. Both runs also applied #5988 (no effect with TP only). The SFT script's decode steps are the same commands as the RL script's but have not been run end to end yet. Not validated on v5p.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.