Skip to content

fix(vllm): interleave prefused MoE wi only for the TP fused kernel - #6225

Open
chiajunglien wants to merge 1 commit into
mainfrom
emma/fuse-moe-weights-interleave-only-for-tp-kernel
Open

chiajunglien wants to merge 1 commit into
mainfrom
emma/fuse-moe-weights-interleave-only-for-tp-kernel

Conversation

@chiajunglien

@chiajunglien chiajunglien commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Follow-up to #5988, from @A9isha's review.

_fuse_moe_weights (model_creation_utils.py) builds the prefused MoE wi from a checkpoint's wi_0/wi_1. It interleaved gate and up per shard by default, and #5988 built [gate | up] only for tpu-inference's EP fused kernel. But the TP fused kernel is the only consumer that splits each shard of the fused axis into gate and up; RoutedMoE's non-fused path splits the whole axis in half. So under vLLM attention with TP > 1, a hash-routed layer, which never runs the fused kernel, got a wrongly split wi. The interleave was also the default when no config is passed.

Changes

  • Inverted the predicate (_fused_moe_kernel_uses_ep → _fused_moe_kernel_uses_tp): interleave only when the TP fused kernel runs the layer (vLLM attention, EP size 1, no hash routing, i.e. no tid2eid), and build [gate | up] otherwise, including when no config is passed.
  • A cross-reference comment where the non-fused path in moe.py splits the prefused wi.
  • The small notes from the review:
    • Both layouts are built in the model's dtype (the [gate | up] buffer is allocated in it). from_pretrained already restores wi_0/wi_1 in the model's dtype, so this only matters for other callers.
    • Logs when a sharded embed axis forces a single-pass build.
    • Both builders raise a ValueError with a message when the halves do not split evenly into the shards.
    • The embed-slices test docstring notes that donation is a no-op on CPU.

The layouts for the TP and EP fused kernels and for non-vLLM attention do not change (non-vLLM attention already gets [gate | up] on main since b89d5ce).

Notes

BUGS: b/558526641

Tests

CPU, 4 virtual devices: model_creation_utils_test: 72 passed, 27 skipped.

  • New: no config → [gate | up]; hash-routed layer under the TP kernel → [gate | up]; both layouts built in the model's dtype; EP with an unsharded fused axis still uses the sliced builder; divisibility errors in both builders; log on a sharded embed axis.
  • The no-config, hash-routed, dtype, divisibility and log tests fail against main's model_creation_utils.py.

No TPU run, since the layouts for the TP and EP fused kernels do not change.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @chiajunglien, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met; no critical defects found. One consistency issue noted between the two builders.

Top issues:

  1. 🟠 [Alignment] The concatenated path now robustly checks divisibility and allocates in the model's dtype to help external callers. However, the fallback interleaved path (_pad_and_chunk) was not updated, meaning it still risks confusing reshape errors on indivisible shapes and returns the checkpoint's original dtype. → Apply the same ValueError check and astype(wi_model.dtype) cast to the interleaved path.
Inferred spec & validation
  • Sources: PR description; code before/after.
  • Problem: Hash-routed MoE layers under vLLM attention incorrectly received a per-shard interleaved wi (only valid for the TP fused MoE kernel) instead of the expected concatenated [gate | up] halves.
Spec item Implementation Validating test
R1 (stated): Interleave wi_0 and wi_1 per shard only when the consumer is the TP fused MoE kernel (vLLM attention, EP size 1, no hash routing). ✅ met TestFusedMoeKernelUsesTp ✅
R2 (stated): For all other paths, including hash-routed layers and when no config is passed, concatenate the halves as `[gate up]`. ✅ met
C1 (stated): Build the concatenated buffer in the model's dtype. ✅ preserved test_concatenated_layout_is_built_in_the_model_dtype ✅
C2 (stated): Raise a ValueError if the MoE halves cannot be evenly padded per shard. ✅ preserved test_rejects_halves_the_shards_do_not_divide ✅

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

_fuse_moe_weights (model_creation_utils.py) builds the prefused MoE `wi`
from a checkpoint's wi_0/wi_1. It interleaved gate and up per shard
([g0|u0|g1|u1|...]) by default and built [gate | up] only for
tpu-inference's expert-parallel fused kernel. But the TP fused kernel is
the only consumer that splits each shard of the fused axis into gate and
up; RoutedMoE's non-fused path splits the whole axis in half. So under
vLLM attention with TP > 1, a hash-routed layer, which never runs the
fused kernel, got a wrongly split `wi`. The interleave was also the
default when no config is passed.

Invert the predicate: interleave only when the TP fused kernel runs the
layer (vLLM attention, an expert parallelism size of 1, and no hash
routing, i.e. no tid2eid), and build [gate | up] otherwise, including
when no config is passed. The layouts for the TP and EP fused kernels
and for non-vLLM attention do not change.

Also, from the review of #5988:
- Build both layouts in the model's dtype rather than the checkpoint's;
  the [gate | up] buffer is allocated in it.
- Log when a sharded embed axis forces a single-pass build.
- Raise a ValueError with a message in both builders when the halves do
  not split evenly into the shards, instead of failing in reshape.
- Comment in moe.py on where the non-fused path's [gate | up] split
  comes from, and note in a test docstring that donation is a no-op on
  CPU.

Bug: b/558526641
@chiajunglien
chiajunglien force-pushed the emma/fuse-moe-weights-interleave-only-for-tp-kernel branch from 069fbbb to a5c9cf4 Compare November 10, 2026 03:15
@chiajunglien chiajunglien removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant