Repository navigation
Tile m3 Qwen3.8 QSA indexer score and top-k scan over key-block chunks - #6407
AusarYao28 wants to merge 1 commit into
Conversation
Qwen3_8QSAIndexer.__call__ materialized the full [B, S, H, M] fp32 raw score tensor and [B, S, M] masked score tensor before jax.lax.top_k even on the return_token_mask=False flash path, creating an O(S^2 / c) HBM bottleneck (~3.2 GiB at S=32K and ~12.4 GiB at S=64K per QSA layer on tpu7x-8). - Add tiled_qsa_top_k in modeling_qwen3_8_flash_next.py scanning key-block chunks of size block_chunk_size (default min(512, block_topk)) with a running stable top-k merge (run_vals before chunk_scores) that preserves bit-identical block selection and tie-breaking while keeping the query sequence axis untouched for context/EP=CP sharding. - Wrap the tiled scan in a zero-residual custom_vjp with shard-local zero barriers so outer scan + jax.checkpoint policies never hoist or stack [num_chunks, B, S, Mc] tensors and ShardMode.EXPLICIT gradients remain zero and properly sharded. - Keep return_token_mask=True (default qsa_attention_kernel="dense") on the untouched dense path so existing compiled HLO is byte-identical. - Add unit tests covering bit-exact parity (unpacked, packed multi-doc with padding, forced zero/negative/duplicate ties, S < token_budget, S % c != 0, B > 1), compiled HLO tiling, partial-tail chunk-boundary negative control, remat residual checks, 4-device sequence-sharded HLO collective checks, and default-off lowered HLO identity (return_token_mask=True and dense QSA).
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a memory-efficient tiling strategy for the Qwen3.8 QSA indexer. By scanning key-block chunks instead of materializing large score tensors, the implementation significantly lowers HBM consumption for long sequences while maintaining bit-exact parity with the existing dense path. The changes include a new tiled kernel, updated indexer logic, and comprehensive unit tests to verify parity, memory savings, and correct sharding behavior. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after, and tests.
- Problem: The
Qwen3_8QSAIndexermaterialized the full[B, S, H, M]raw score tensor and[B, S, M]masked score tensor beforejax.lax.top_kon thereturn_token_mask=Falseflash path, creating anO(S^2 / c)HBM bottleneck.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Tile key-block dimension to compute top-k QSA block scores via scanning chunks when return_token_mask=False. |
✅ | test_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat ✅ |
| R2 (stated): Selected block indices and tie-breaking must be bit-identical to the dense path. | ✅ | test_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat ✅ |
R3 (stated): Wrap in zero-residual @jax.custom_vjp with shard-local zero barriers to prevent hoisting large tensors in outer lax.scan + jax.checkpoint. |
✅ | test_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat ✅ |
| C1 (stated): Default dense path untouched and byte-identical in lowered StableHLO. | ✅ preserved | test_qsa_tiled_indexer_default_off_lowered_hlo_identity ✅ |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Description
Depends on #6283 (stacked on
yaoyuchen/m3-qwen38-qsa-block-sparse; review the top commit only).Qwen3_8QSAIndexer.__call__previously materialized the full[B, S, H, M]float32 raw score tensor and[B, S, M]masked score tensor beforejax.lax.top_keven on thereturn_token_mask=Falseflash path (M = ceil(S / block_size)), creating anO(S^2 / c)HBM bottleneck (~3.2 GiB atS=32Kand ~12.4 GiB atS=64Kper QSA layer ontpu7x-8).This PR tiles the key-block dimension in
src/maxtext/m3/models/qwen3_8_flash_next/modeling_qwen3_8_flash_next.py:tiled_qsa_top_kscanning key-block chunks of sizeblock_chunk_size(defaultmin(512, block_topk)) whenreturn_token_mask=Falseandnum_blocks > block_chunk_size, with a running stable top-k merge (run_valsconcatenated beforechunk_scores,-infpadded tail) so selected block indices and tie-breaking are bit-identical to the dense path while leaving(B, S)unsplit for context/EP=CP sharding.@jax.custom_vjpwith shard-local zerooptimization_barriers so outerlax.scan+jax.checkpointpolicies never hoist[num_chunks, B, S, Mc]tensors andShardMode.EXPLICITgradients remain properly sharded.qsa_attention_kernel="dense"/return_token_mask=True) untouched and byte-identical in lowered StableHLO.Tests
tests/unit/m3/qwen3_8_flash_next/modeling_qwen3_8_flash_next_test.py -k tiled_indexer: 3 passed (covers bit-exact parity across unpacked, packed multi-document with padding, forced zero/negative/duplicate ties,S < token_budget,S % c != 0,B > 1; compiled HLO tiling; partial-tail negative control; remat residuals; 4-device sequence-sharded HLO collective checks; and committed default-off lowered StableHLO identity).tests/unit/m3/qwen3_8_flash_next/: 65 passed, 13 skipped.-jnp.infto0.0failstest_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat(30/640index mismatches).[compiled AOT], isolatedQwen3_8QSAIndexer, unpackedB_dev=1):S=16K:853 -> 406 MiB(-52.4%)S=32K:3.20 -> 0.73 GiB(-77.2%)S=64K:12.38 -> 1.46 GiB(-88.2%,8.5xreduction)(fsdp=2, context=4)tiled indexer emits0all-gatherand0all-reducecollectives.Qwen3_8SparseAttentionfwd+bwd:S=8K1.42 -> 1.24 GiB,S=64K12.41 -> 11.07 GiB(-10.8%).pyink(2-space, 122 cols),pylint(10.00/10), andcodespellclean.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.