Repository navigation
Add tiled block-sparse FlashAttention with 32-bit packed bitmap mask for m3 Qwen3.8 QSA - #6283
AusarYao28 wants to merge 1 commit into
Conversation
|
🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a memory-efficient tiled block-sparse FlashAttention kernel for Qwen3.8 Sparse Attention. By replacing dense mask materialization with a compact 32-bit packed bitmap and implementing a tiled online-softmax approach, the changes allow for training on much longer sequences while maintaining numerical parity with the existing dense implementation. The solution includes robust sharding support and custom VJP backward passes to optimize memory footprint during training. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
📋 Review Summary
Verdict: High-quality implementation; FlashAttention-2 custom VJP and 32-bit packed bitmap mask cleanly resolve the QSA OOM while preserving numerical parity and sharding.
Top issues:
-
[Perf]
_compute_qsa_doc_starteliminates$O(S^2)$ masks in attention, butQwen3_8QSAIndexer.__call__still evaluatessame_doc = (segment_ids[:, :, None] == segment_ids[:, None, :])to computedoc_starteven whenreturn_token_mask=False. → InQwen3_8QSAIndexer, reuse_compute_qsa_doc_start(segment_ids)whennot return_token_maskto avoid$O(B \cdot S^2)$ boolean allocations during multi-doc indexing (see inline comment onmodeling_qwen3_8_flash_next.py). -
[Tests] Section 4 of
test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controlsguards 4-device mesh sharding withif len(jax.devices()) >= 4:, silently passing without execution on single-device environments. → Split the multi-device check into a dedicated test method withif len(jax.devices()) < 4: self.skipTest(...)matching lines 1414 and 2018 (see inline comment onmodeling_qwen3_8_flash_next_test.py).
Inferred spec & validation
- Sources: PR #6283 description; code before/after; internal ticket b/580120934 (not readable here).
-
Problem: For sequence lengths exceeding the token budget (
$S > 2048$ ),Qwen3_8SparseAttentionpreviously materialized dense$O(S^2)$ attention matrices in HBM ($36\text{ GiB}$ at$S=8192$ , OOM at$97.5\text{ GiB}$ at$S=16384$ ), whileqsa_attention_kernel="flash"rejected$S > 2048$ with aValueError.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Implement tiled block-sparse FlashAttention (flash_block_sparse_attention) for |
✅ met |
modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation ✅ |
R2 (stated): Compact 32-bit uint32 packed block bitmap via deduplicated scatter-add without materializing booleans |
✅ met |
modeling_qwen3_8_flash_next_test.py::test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls (section 1) ✅ |
| R3 (stated): Tiled online-softmax forward with causal tile skipping, GQA head broadcasting, and VMEM mask unpacking | ✅ met |
modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation (unpacked/mask_2d/packed_seg/both) ✅ |
| R4 (stated): FlashAttention-2 custom VJP backward recomputing tile probabilities with compiler optimization barrier | ✅ met |
modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation (value_and_grad), section 5 (jaxpr check) ✅ |
R5 (stated): Multi-document isolation (segment_ids), 2D attention_mask padding, and non-divisible tail block handling |
✅ met |
modeling_qwen3_8_flash_next_test.py::test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls (sections 2, 3) ✅ |
C1 (stated): Backward compatibility; keep qsa_attention_kernel="dense" as default in types.py
|
✅ preserved |
modeling_qwen3_8_flash_next_test.py::test_kernel_path_flags_match_defaults_end_to_end ✅ |
C2 (stated): Forward and backward numerical parity against dense reference in float32 and bfloat16
|
✅ preserved |
modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation ✅ |
C3 (stated): Preservation of (data, tensor) sharding across tile transformations without communication inside kernel |
✅ preserved |
modeling_qwen3_8_flash_next_test.py::test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls (section 4) ✅ |
| self.assertGreater(float(jnp.max(jnp.abs(out_leaked[:, len1:, :] - out_cat[:, len1:, :]))), 0.05) | ||
|
|
||
| # 4. 4-device CPU sharded parity (data=2, tensor=2) for s > token_budget. | ||
| if len(jax.devices()) >= 4: |
There was a problem hiding this comment.
🟡 Guarding multi-device verification with if len(jax.devices()) >= 4: causes this sub-test to pass silently without executing on standard single-device CPU test runs. Consider splitting section 4 into a dedicated test method test_qsa_flash_block_sparse_4device_sharded_parity with if len(jax.devices()) < 4: self.skipTest(...), following the pattern in lines 1414 and 2018 so that test skips are explicitly reported by unittest.
There was a problem hiding this comment.
Fixed: extracted section 4 into a dedicated test_qsa_flash_block_sparse_4device_sharded_parity test method that calls self.skipTest("Requires >= 4 JAX devices") when len(jax.devices()) < 4.
| return token_mask, selected_block_indices | ||
|
|
||
|
|
||
| def _compute_qsa_doc_start(segment_ids: jax.Array) -> jax.Array: |
There was a problem hiding this comment.
🟡 _compute_qsa_doc_start computes document start offsets in O(B * S log S) without O(S^2) masks and is used in flash_block_sparse_attention, but Qwen3_8QSAIndexer.__call__ (line 660) still evaluates same_doc = (segment_ids[:, :, None] == segment_ids[:, None, :]) & (segment_ids[:, :, None] != 0) to derive doc_start even when return_token_mask=False. For long multi-document sequences, reusing _compute_qsa_doc_start in Qwen3_8QSAIndexer when not return_token_mask will eliminate the remaining O(B * S^2) boolean allocation in the indexer.
There was a problem hiding this comment.
Fixed: moved _compute_qsa_doc_start above Qwen3_8QSAIndexer and used it (with a fast jnp.zeros path when unpacked) to compute doc_start in Qwen3_8QSAIndexer.__call__, deferring the O(B * S^2) same_doc allocation until return_token_mask=True.
ab11fad to
06c7dae
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details. |
72a612e to
06b61f1
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found. The block-sparse FlashAttention implementation elegantly eliminates the O(B*S^2) mask materialization, handles tail blocks and multi-document packing accurately, and correctly prevents loop-carried residuals in the custom VJP.
Inferred spec & validation
- Sources: PR description, b/580120934 (inferred from description context as Buganizer is internal), code before/after.
- Problem: When sequence length > 2048,
Qwen3_8SparseAttentionmaterialized a dense[B, S, S]boolean mask and dense logits, causing massive HBM usage and OOM on TPU atS=16384.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Pack block mask into 32-bit uint32 bitmap. |
✅ met | test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅ |
| R2 (stated): Tiled online-softmax forward skipping causal future tiles. | ✅ met | test_flash_attention_kernel_parity_and_validation ✅ |
R3 (stated): Custom VJP backward saving only required activations to avoid O(B*S^2) memory. |
✅ met | test_flash_attention_kernel_parity_and_validation ✅ |
| R4 (stated): Prevent 6D/7D branch residuals from being stacked by outer scan remat. | ✅ met | test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅ |
C1 (stated): Fallback to Tokamax Splash Attention when S <= token_budget and keep dense as default. |
✅ preserved | test_flash_attention_kernel_parity_and_validation ✅ |
| C2 (inferred): Retain precision and numerical equivalence with dense fallback path in bf16 and fp32. | ✅ preserved | test_flash_attention_kernel_parity_and_validation ✅ |
|
Hi @AusarYao28, thanks for the heads-up! 1. Rebase of #6274 onto #6205 (
|
e00130a to
a6b500e
Compare
06b61f1 to
7828327
Compare
a6b500e to
8d7500a
Compare
7828327 to
7fea1aa
Compare
|
Thanks so much for verifying 4D
|
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found. The implementation of the tiled block-sparse FlashAttention kernel is mathematically sound, highly optimized, and exceptionally well-tested.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem:
Qwen3_8SparseAttentionmaterialized dense masks and attention logits in HBM when sequence length exceeded the token budget, leading to OOMs and poor MXU utilization due to independent KV gather.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Implement a tiled block-sparse FlashAttention kernel for s > token_budget (qsa_attention_kernel="flash"). |
✅ | test_flash_attention_kernel_parity_and_validation ✅ |
R2 (stated): Pack the per-query selected_block_indices into a carry-free 32-bit uint32 bitmap. |
✅ | test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅ |
R3 (stated): Preserve memory via FlashAttention-2 LSE @jax.custom_vjp that saves O(B * S) memory. |
✅ | test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅ |
R4 (stated): Resolve query-side length_axes via jax.shard_map and handle GQA tensor head sharding. |
✅ | test_qsa_flash_block_sparse_sequence_sharding_and_custom_strategy ✅ |
C1 (stated): Keep qsa_attention_kernel="dense" as the default configuration. |
✅ preserved | modeling_qwen3_8_flash_next_test.py ✅ |
C2 (stated): Fallback to Tokamax Splash Attention when s <= token_budget under the flash kernel. |
✅ preserved | test_flash_attention_kernel_parity_and_validation ✅ |
8d7500a to
9f3137a
Compare
7fea1aa to
3957c4b
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: Dense QSA attention materialize
[B, S, S]masks and[B, H_q, S, S]FP32 attention tensors causing OOM for sequences > 2048, with poor MXU utilization during local KV gathering.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Implement block-sparse FlashAttention path when S > index_select_budget. | ✅ | test_flash_attention_kernel_parity_and_validation ✅ |
| R2 (stated): Pack QSA selected block masks into a 32-bit uint32 carry-free bitmap. | ✅ | test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅ |
| R3 (stated): LSE-based custom_vjp to save O(B * S) memory and zero 6D/7D branch residuals. | ✅ | test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅ |
R4 (stated): Preserve data, tensor, and context sharding without Q all-gathers. |
✅ | test_qsa_flash_block_sparse_sequence_sharding_and_custom_strategy ✅ |
C1 (stated): Maintain backward compatibility with qsa_attention_kernel="dense" as default. |
✅ preserved | test_flash_attention_kernel_parity_and_validation ✅ |
…for m3 Qwen3.8 QSA
9f3137a to
b9a5815
Compare
3957c4b to
69058e7
Compare
|
🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details. |
Add tiled block-sparse FlashAttention with 32-bit packed bitmap mask for m3 Qwen3.8 QSA
Description
Depends on #6205; review the top commit only.
When sequence length exceeds the Qwen3.8 QSA token budget (
S > 2048),Qwen3_8SparseAttentionpreviously materialized a dense[B, S, S]boolean mask and dense[B, H_q, S, S]float32attention logits/probabilities in HBM (36.00 GiBof[2, 24, 8192, 8192]buffers atS=8192andOOMat97.50 GiBsingle-layer temp HBM atB=2, S=16384). BecauseQwen3_8QSAIndexerselectsk_blk = 512blocks (2048key tokens) independently per query token, per-query KV gather materializes[B, S, 2052, H_kv, D](68.85 GBforward /206.56 GBforward+backward atB=2, S=8192) with onlyM = G = 12rows per MXU matmul (4.7%of a256 x 256TPU7x MXU systolic tile), while query-chunk union gather covers85%-100%of all blocks in the sequence by the birthday paradox.This PR implements a tiled block-sparse FlashAttention path (
qsa_attention_kernel="flash") forQwen3_8SparseAttentionwhenS > token_budget:pack_qsa_block_mask_32bit): Packs per-queryselected_block_indices([B, S, k_blk]int32, with-1for invalid slots) into a compact[B, S, num_blocks // 32]uint32bitmap (4.19 MBatB=2, S=8192) via deduplicated unsigned scatter-add without materializing[B, S, num_blocks]booleans.Qwen3_8SparseAttention.flash_block_sparse_attention): Streams contiguous un-gatheredK, Vtiles (B_kv = 1024) against query tiles (B_q = 512) reshaped as[num_q_tiles, B, B_q, H_kv, G, D](M = B_q * G = 6144MXU rows per KV head,100%MXU tile utilization, zero12xKV repeat), skips strictly future KV tiles (i_kv * B_kv > (i_q + 1) * B_q - 1) viajax.lax.cond, and unpacks[B, B_q, B_kv]tile masks on the fly in VMEM from8uint32words per KV tile (ORing current-block self-attention and ANDing causal, 2D padding, and packedsegment_idsdocument masks).@jax.custom_vjpbackward: Saves only(q, k, v, packed_mask, out, lse)(O(B * S)memory;lseis3 MBatB=2, S=8192) and recomputes tile probabilities on the fly in backward. Tiesstop_gradientinteger masks and loop tile indices to dynamicq, k, vinputs via a zero-valuedjax.lax.optimization_barrierscalar so JAX's_scan_known_hoisting/_cond_partial_evalunder outerjax.lax.scan+remat_policy=custom(save_only_these_names("query_proj")) never hoists or stacks 6D/7D[num_q_tiles, num_kv_tiles, B, B_q, H_kv, G, B_kv]branch residuals (0.00 GiBstacked tile residuals in full-model HLO), and preserves(data, tensor)sharding (h_kv = 1per TP rank ontp=2) across 6D/5D tile reshapes.length_axesjax.shard_map: Resolvesbatch,heads(tensor+tensor_sequence, with GQAk/vrepeat whenhead_shards > num_kv_heads), andlength(contextandEP=CPexpert) viaself.sharding, packs bitmaps viajax.vmap(jax.vmap(...)), and scanslocal_q_tiles = s_q_loc // b_qper sequence shard with zero queryall-gathers.qsa_attention_kernel="dense"as the default config setting and dispatchesqsa_attention_kernel="flash"through Tokamax Splash Attention whenS <= token_budgetandflash_block_sparse_attentionwhenS > token_budget.FIXES: b/580120934
Tests
4-CPU Unit & Sharding Tests (
tests/unit/m3/qwen3_8_flash_next/modeling_qwen3_8_flash_next_test.py,tests/unit/m3/qwen3_8_rope_precision_test.py):JAX_PLATFORMS=cpu XLA_FLAGS="--xla_force_host_platform_device_count=4" pytest -q tests/unit/m3/qwen3_8_flash_next/modeling_qwen3_8_flash_next_test.py -k "qsa" tests/unit/m3/qwen3_8_rope_precision_test.py:8 passed, 2 skipped, 13 subtests passed.nnx.value_and_grad) numerical parity betweenqsa_attention_kernel="flash"andqsa_attention_kernel="dense"acrossunpacked,mask_2d,packed_seg, andboth_mask_and_segin bothfloat32(max_diff <= 1.5e-5) andbfloat16, multi-word 32-bit bitmap packing with duplicate indices and-1padding, non-multiple-of-compress_ratiotail blocks (S=70, c=4), multi-word fast unpacked path (S=256, B_kv=128), packed 2-document isolation (packed_out == concat(unpacked_doc1, unpacked_doc2)in forward and backward),Qwen3_8QSAIndexer(..., return_token_mask=False)bit-identicalselected_block_indicesparity with zeroO(B * S^2)bool[B, S, S]allocation, 4-device(data=2, tensor=2)mesh sharding (verifying per-shardh_kv = 1in compiled HLO, with explicitself.skipTestwhen< 4devices), and outerjax.lax.scan+jax.checkpoint(save_only_these_names("query_proj"),nothing_saveable,everything_saveable) residual check asserting zero 6D/7D stacked tile buffers and zerobool[B, S, S]indexer masks.packed_maskor droppingsegment_idsacross packed documents producesmax_abs_diff > 0.05against the exact output, proving the tests fail if sparse block masking or packed document isolation is broken.Module-Level
Qwen3_8SparseAttention(Indexer + Projections + Attention, Forward + Backward)tpu7x-8AOT HBM ([compiled AOT tpu7x-8],data=4, tensor=2, per-deviceB=1, globalB=4):Sqsa_attention_kernel[compiled AOT tpu7x-8])[compiled AOT tpu7x-8])denseS = 4096densePASS2.773 GiB3.038 GiBflashPASS0.712 GiB0.976 GiB-2.061 GiB(-74.3%)S = 8192densePASS10.080 GiB10.384 GiBflashPASS1.462 GiB1.766 GiB-8.618 GiB(-85.5%)S = 16384densePASS38.323 GiB38.705 GiBflashPASS2.824 GiB3.206 GiB-35.499 GiB(-92.6%,12.1xpeak reduction)qwen3.8-flash-nexttrain_compile.pyAOT ontpu7x-8(ici_fsdp_parallelism=4, ici_tensor_parallelism=2, remat_policy=custom, decoder_layer_input=offload) ([compiled AOT tpu7x-8]):Sqsa_attention_kernel[2, 24, S, S]BuffersPeak memoryTotal bytes(preallocated-temp)TRAIN-KERNEL DEVICE PEAK(Temp)S = 4096dense0(0.00 GiB)4 x 3.00 = 12.00 GiB62.28 GiB83.30 GiB(55.44 GiB)112.21 GiB(84.36 GiB)339.9 MiBflash(06c7daec4)0(0.00 GiB)0(0.00 GiB)62.54 GiB83.95 GiB(56.09 GiB)113.67 GiB(85.81 GiB)187.4 MiB(-44.9%)S = 8192dense0(0.00 GiB)2 x 12.00 + 2 x 6.00 = 36.00 GiB73.73 GiB79.31 GiB(51.45 GiB)88.60 GiB(60.75 GiB)509.9 MiBflash(06c7daec4)0(0.00 GiB)0(0.00 GiB)74.51 GiB76.20 GiB(-3.11 GiB)82.16 GiB(54.31 GiB,-6.44 GiB/-10.6%Temp)187.4 MiB(-63.2%)(In
after_optimizations-live-range.txt, the global live-rangePeak memoryof~62.3-62.5 GiBatS=4096and~73.7-74.5 GiBatS=8192occurs inside the 512-expert MoE backward passtgmm_v2+25.00 GiBexpert weight-gradientall-reduce+27.86 GiBmodel/optimizer state, not inside QSA.)tpu7x-8(2x2x1,bodaborg-tpu7x-nap) End-to-End Training A/B ([measured tpu7x-8]):f2d94500, which executes the identical inner tiled FlashAttention loop prior toab11fad28's-26.94 GiBremat_policy=custombranch-residual fix documented in the AOT table above.)4L qwen3.8-flash-next,tpu7x-8,fsdp=4, tp=2)p50Step Time ([measured tpu7x-8])TFLOP/s/dev)densegrad_normqsa_4L_s4096_dense(run 1 / run 2)4383.0 / 4382.5 ms6.5631.00x12.5459.7190.626928qsa_4L_s4096_flash1062.0 ms27.0784.13xfaster12.5459.7180.626921qsa_4L_s8192_dense(run 1 / run 2)15086.0 / 15085.0 ms3.8251.00x12.54610.1130.482197qsa_4L_s8192_flash1635.5 ms35.2799.22xfaster12.54610.1130.482203ab11fad28) NativeFSDP=2, EP=4tpu7x-8(2x2x1,bodaborg-tpu7x-nap) Measured Results ([measured tpu7x-8]):m3-qsa-ab11-2x2x1-bench-0-0-5nncp(measured with nativeFSDP=2, EP=4sharding withoutsharding_toleranceoverride):S = 4096:2397.75 ms -> 732.00 ms(3.28xfaster, throughput12.00 -> 39.26 TFLOP/s/dev, runtime compiled HBM43.62 -> 43.44 GiB[-0.18 GiB], live allocated peak16.49 -> 16.50 GiB, Step 14 loss10.204 -> 10.204[0.000diff], Step 0raw_grad_normrelative diff4.60e-6).S = 8192:7979.75 ms -> 1261.50 ms(6.33xfaster, throughput7.23 -> 45.85 TFLOP/s/dev, runtime compiled HBM63.25 -> 63.47 GiB[0.00 GiBraw peak delta,56.32 -> 56.32 GiB], live allocated peak16.54 -> 16.55 GiB, Step 14 loss10.530 -> 10.530[0.000diff], Step 0raw_grad_normrelative diff8.09e-6).4L qwen3.8-flash-next,tpu7x-8, nativefsdp=2, ep=4)p50Step Time ([measured tpu7x-8])TFLOP/s/dev)denseS = 4096dense (2 runs)2397.75 ms(2398.0 / 2397.5)12.001.00x10.20443.62 GiB(raw 37.74)16.49 GiBS = 4096flash (1 run)732.00 ms39.263.28x10.204(0.000diff)43.44 GiB(-0.18 GiB)16.50 GiBS = 8192dense (2 runs)7979.75 ms(7979.0 / 7980.5)7.231.00x10.53063.25 GiB(raw 56.32)16.54 GiBS = 8192flash (steps 0..17)1261.50 ms45.856.33x10.530(0.000diff)63.47 GiB(0.00 GiBraw delta)16.55 GiBChecklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.