Skip to content

Tile m3 Qwen3.8 QSA indexer score and top-k scan over key-block chunks - #6407

Open
AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-qwen38-qsa-block-sparsefrom
yaoyuchen/m3-qwen38-qsa-tiled-indexer
Open

AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-qwen38-qsa-block-sparsefrom
yaoyuchen/m3-qwen38-qsa-tiled-indexer

Conversation

@AusarYao28

Copy link
Copy Markdown
Collaborator

Description

Depends on #6283 (stacked on yaoyuchen/m3-qwen38-qsa-block-sparse; review the top commit only).

Qwen3_8QSAIndexer.__call__ previously materialized the full [B, S, H, M] float32 raw score tensor and [B, S, M] masked score tensor before jax.lax.top_k even on the return_token_mask=False flash path (M = ceil(S / block_size)), creating an O(S^2 / c) HBM bottleneck (~3.2 GiB at S=32K and ~12.4 GiB at S=64K per QSA layer on tpu7x-8).

This PR tiles the key-block dimension in src/maxtext/m3/models/qwen3_8_flash_next/modeling_qwen3_8_flash_next.py:

  • Adds tiled_qsa_top_k scanning key-block chunks of size block_chunk_size (default min(512, block_topk)) when return_token_mask=False and num_blocks > block_chunk_size, with a running stable top-k merge (run_vals concatenated before chunk_scores, -inf padded tail) so selected block indices and tie-breaking are bit-identical to the dense path while leaving (B, S) unsplit for context/EP=CP sharding.
  • Wraps the tiled scan in a zero-residual @jax.custom_vjp with shard-local zero optimization_barriers so outer lax.scan + jax.checkpoint policies never hoist [num_chunks, B, S, Mc] tensors and ShardMode.EXPLICIT gradients remain properly sharded.
  • Leaves the default dense path (qsa_attention_kernel="dense" / return_token_mask=True) untouched and byte-identical in lowered StableHLO.

Tests

  • CPU unit tests (4-device host mesh, serial):
    • tests/unit/m3/qwen3_8_flash_next/modeling_qwen3_8_flash_next_test.py -k tiled_indexer: 3 passed (covers bit-exact parity across unpacked, packed multi-document with padding, forced zero/negative/duplicate ties, S < token_budget, S % c != 0, B > 1; compiled HLO tiling; partial-tail negative control; remat residuals; 4-device sequence-sharded HLO collective checks; and committed default-off lowered StableHLO identity).
    • Full tests/unit/m3/qwen3_8_flash_next/: 65 passed, 13 skipped.
    • Negative control: changing the padded tail fill from -jnp.inf to 0.0 fails test_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat (30/640 index mismatches).
  • TPU7x-8 AOT memory ([compiled AOT], isolated Qwen3_8QSAIndexer, unpacked B_dev=1):
    • S=16K: 853 -> 406 MiB (-52.4%)
    • S=32K: 3.20 -> 0.73 GiB (-77.2%)
    • S=64K: 12.38 -> 1.46 GiB (-88.2%, 8.5x reduction)
    • (fsdp=2, context=4) tiled indexer emits 0 all-gather and 0 all-reduce collectives.
    • Isolated Qwen3_8SparseAttention fwd+bwd: S=8K 1.42 -> 1.24 GiB, S=64K 12.41 -> 11.07 GiB (-10.8%).
  • Pre-commit linters: pyink (2-space, 122 cols), pylint (10.00/10), and codespell clean.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

Qwen3_8QSAIndexer.__call__ materialized the full [B, S, H, M] fp32 raw score
tensor and [B, S, M] masked score tensor before jax.lax.top_k even on the
return_token_mask=False flash path, creating an O(S^2 / c) HBM bottleneck
(~3.2 GiB at S=32K and ~12.4 GiB at S=64K per QSA layer on tpu7x-8).

- Add tiled_qsa_top_k in modeling_qwen3_8_flash_next.py scanning key-block
  chunks of size block_chunk_size (default min(512, block_topk)) with a
  running stable top-k merge (run_vals before chunk_scores) that preserves
  bit-identical block selection and tie-breaking while keeping the query
  sequence axis untouched for context/EP=CP sharding.
- Wrap the tiled scan in a zero-residual custom_vjp with shard-local zero
  barriers so outer scan + jax.checkpoint policies never hoist or stack
  [num_chunks, B, S, Mc] tensors and ShardMode.EXPLICIT gradients remain
  zero and properly sharded.
- Keep return_token_mask=True (default qsa_attention_kernel="dense") on the
  untouched dense path so existing compiled HLO is byte-identical.
- Add unit tests covering bit-exact parity (unpacked, packed multi-doc with
  padding, forced zero/negative/duplicate ties, S < token_budget, S % c != 0,
  B > 1), compiled HLO tiling, partial-tail chunk-boundary negative control,
  remat residual checks, 4-device sequence-sharded HLO collective checks, and
  default-off lowered HLO identity (return_token_mask=True and dense QSA).
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a memory-efficient tiling strategy for the Qwen3.8 QSA indexer. By scanning key-block chunks instead of materializing large score tensors, the implementation significantly lowers HBM consumption for long sequences while maintaining bit-exact parity with the existing dense path. The changes include a new tiled kernel, updated indexer logic, and comprehensive unit tests to verify parity, memory savings, and correct sharding behavior.

Highlights

  • Memory Optimization: Implemented a tiled top-k scanning approach for QSA indexer, reducing HBM memory usage by up to 8.5x for large sequence lengths (S=64K).
  • Tiled Scan Implementation: Added tiled_qsa_top_k which processes key-block chunks, ensuring bit-identical results to the dense path while avoiding full score tensor materialization.
  • JAX Integration: Wrapped the tiled scan in a @jax.custom_vjp with optimization barriers to maintain correct sharding and prevent tensor hoisting during checkpointing.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 17, 2026
@AusarYao28 AusarYao28 removed the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 17, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after, and tests.
  • Problem: The Qwen3_8QSAIndexer materialized the full [B, S, H, M] raw score tensor and [B, S, M] masked score tensor before jax.lax.top_k on the return_token_mask=False flash path, creating an O(S^2 / c) HBM bottleneck.
Spec item Implementation Validating test
R1 (stated): Tile key-block dimension to compute top-k QSA block scores via scanning chunks when return_token_mask=False. ✅ test_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat ✅
R2 (stated): Selected block indices and tie-breaking must be bit-identical to the dense path. ✅ test_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat ✅
R3 (stated): Wrap in zero-residual @jax.custom_vjp with shard-local zero barriers to prevent hoisting large tensors in outer lax.scan + jax.checkpoint. ✅ test_qsa_tiled_indexer_parity_ties_padding_boundary_and_remat ✅
C1 (stated): Default dense path untouched and byte-identical in lowered StableHLO. ✅ preserved test_qsa_tiled_indexer_default_off_lowered_hlo_identity ✅

@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 1.21951% with 81 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
.../qwen3_8_flash_next/modeling_qwen3_8_flash_next.py 1.21% 81 Missing ⚠️

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant