Repository navigation
fix(vllm): interleave prefused MoE wi only for the TP fused kernel - #6225
Open
chiajunglien wants to merge 1 commit into
Open
chiajunglien wants to merge 1 commit into
chiajunglien wants to merge 1 commit into
Conversation
chiajunglien
requested review from
NuojCheng,
RissyRan,
abhinavclemson,
darisoy,
gagika,
gobbleturk,
igorts-git,
khatwanimohit,
michelle-yooh,
shralex,
shuningjin and
zxhe-sean
as code owners
November 10, 2026 02:48
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
chiajunglien
requested review from
A9isha,
SurbhiJainUSC,
aireenmei,
bvandermoon,
dipannita08,
hengtaoguo,
jiangjy1982,
vipannalla and
xibinliu
as code owners
November 10, 2026 02:48
Contributor
|
🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Contributor
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met; no critical defects found. One consistency issue noted between the two builders.
Top issues:
- 🟠 [Alignment] The concatenated path now robustly checks divisibility and allocates in the model's dtype to help external callers. However, the fallback interleaved path (
_pad_and_chunk) was not updated, meaning it still risks confusingreshapeerrors on indivisible shapes and returns the checkpoint's original dtype. → Apply the sameValueErrorcheck andastype(wi_model.dtype)cast to the interleaved path.
Inferred spec & validation
- Sources: PR description; code before/after.
- Problem: Hash-routed MoE layers under vLLM attention incorrectly received a per-shard interleaved
wi(only valid for the TP fused MoE kernel) instead of the expected concatenated[gate | up]halves.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Interleave wi_0 and wi_1 per shard only when the consumer is the TP fused MoE kernel (vLLM attention, EP size 1, no hash routing). |
✅ met | TestFusedMoeKernelUsesTp ✅ |
| R2 (stated): For all other paths, including hash-routed layers and when no config is passed, concatenate the halves as `[gate | up]`. | ✅ met |
| C1 (stated): Build the concatenated buffer in the model's dtype. | ✅ preserved | test_concatenated_layout_is_built_in_the_model_dtype ✅ |
| C2 (stated): Raise a ValueError if the MoE halves cannot be evenly padded per shard. | ✅ preserved | test_rejects_halves_the_shards_do_not_divide ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
_fuse_moe_weights (model_creation_utils.py) builds the prefused MoE `wi` from a checkpoint's wi_0/wi_1. It interleaved gate and up per shard ([g0|u0|g1|u1|...]) by default and built [gate | up] only for tpu-inference's expert-parallel fused kernel. But the TP fused kernel is the only consumer that splits each shard of the fused axis into gate and up; RoutedMoE's non-fused path splits the whole axis in half. So under vLLM attention with TP > 1, a hash-routed layer, which never runs the fused kernel, got a wrongly split `wi`. The interleave was also the default when no config is passed. Invert the predicate: interleave only when the TP fused kernel runs the layer (vLLM attention, an expert parallelism size of 1, and no hash routing, i.e. no tid2eid), and build [gate | up] otherwise, including when no config is passed. The layouts for the TP and EP fused kernels and for non-vLLM attention do not change. Also, from the review of #5988: - Build both layouts in the model's dtype rather than the checkpoint's; the [gate | up] buffer is allocated in it. - Log when a sharded embed axis forces a single-pass build. - Raise a ValueError with a message in both builders when the halves do not split evenly into the shards, instead of failing in reshape. - Comment in moe.py on where the non-fused path's [gate | up] split comes from, and note in a test docstring that donation is a no-op on CPU. Bug: b/558526641
chiajunglien
force-pushed
the
emma/fuse-moe-weights-interleave-only-for-tp-kernel
branch
from
November 10, 2026 03:15
069fbbb to
a5c9cf4
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #5988, from @A9isha's review.
_fuse_moe_weights(model_creation_utils.py) builds the prefused MoEwifrom a checkpoint'swi_0/wi_1. It interleaved gate and up per shard by default, and #5988 built[gate | up]only for tpu-inference's EP fused kernel. But the TP fused kernel is the only consumer that splits each shard of the fused axis into gate and up;RoutedMoE's non-fused path splits the whole axis in half. So under vLLM attention with TP > 1, a hash-routed layer, which never runs the fused kernel, got a wrongly splitwi. The interleave was also the default when no config is passed.Changes
_fused_moe_kernel_uses_ep→_fused_moe_kernel_uses_tp): interleave only when the TP fused kernel runs the layer (vLLM attention, EP size 1, no hash routing, i.e. notid2eid), and build[gate | up]otherwise, including when no config is passed.moe.pysplits the prefusedwi.[gate | up]buffer is allocated in it).from_pretrainedalready restoreswi_0/wi_1in the model's dtype, so this only matters for other callers.ValueErrorwith a message when the halves do not split evenly into the shards.The layouts for the TP and EP fused kernels and for non-vLLM attention do not change (non-vLLM attention already gets
[gate | up]on main since b89d5ce).Notes
_fuse_moe_weights, asfrom_pretraineddoes, so they do not depend on theconfig=Nonedefault, and the two PRs can merge in either order.BUGS: b/558526641
Tests
CPU, 4 virtual devices:
model_creation_utils_test: 72 passed, 27 skipped.[gate | up]; hash-routed layer under the TP kernel →[gate | up]; both layouts built in the model's dtype; EP with an unsharded fused axis still uses the sliced builder; divisibility errors in both builders; log on a sharded embed axis.model_creation_utils.py.No TPU run, since the layouts for the TP and EP fused kernels do not change.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.