Skip to content

Add tiled block-sparse FlashAttention with 32-bit packed bitmap mask for m3 Qwen3.8 QSA - #6283

Open
AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-host-offload-hbm-telemetryfrom
yaoyuchen/m3-qwen38-qsa-block-sparse
Open

AusarYao28 wants to merge 1 commit into
yaoyuchen/m3-host-offload-hbm-telemetryfrom
yaoyuchen/m3-qwen38-qsa-block-sparse

Conversation

@AusarYao28

@AusarYao28 AusarYao28 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Add tiled block-sparse FlashAttention with 32-bit packed bitmap mask for m3 Qwen3.8 QSA

Description

Depends on #6205; review the top commit only.

When sequence length exceeds the Qwen3.8 QSA token budget (S > 2048), Qwen3_8SparseAttention previously materialized a dense [B, S, S] boolean mask and dense [B, H_q, S, S] float32 attention logits/probabilities in HBM (36.00 GiB of [2, 24, 8192, 8192] buffers at S=8192 and OOM at 97.50 GiB single-layer temp HBM at B=2, S=16384). Because Qwen3_8QSAIndexer selects k_blk = 512 blocks (2048 key tokens) independently per query token, per-query KV gather materializes [B, S, 2052, H_kv, D] (68.85 GB forward / 206.56 GB forward+backward at B=2, S=8192) with only M = G = 12 rows per MXU matmul (4.7% of a 256 x 256 TPU7x MXU systolic tile), while query-chunk union gather covers 85%-100% of all blocks in the sequence by the birthday paradox.

This PR implements a tiled block-sparse FlashAttention path (qsa_attention_kernel="flash") for Qwen3_8SparseAttention when S > token_budget:

  • Carry-free 32-bit packed block bitmap (pack_qsa_block_mask_32bit): Packs per-query selected_block_indices ([B, S, k_blk] int32, with -1 for invalid slots) into a compact [B, S, num_blocks // 32] uint32 bitmap (4.19 MB at B=2, S=8192) via deduplicated unsigned scatter-add without materializing [B, S, num_blocks] booleans.
  • Tiled online-softmax forward with causal tile skipping and GQA grouping (Qwen3_8SparseAttention.flash_block_sparse_attention): Streams contiguous un-gathered K, V tiles (B_kv = 1024) against query tiles (B_q = 512) reshaped as [num_q_tiles, B, B_q, H_kv, G, D] (M = B_q * G = 6144 MXU rows per KV head, 100% MXU tile utilization, zero 12x KV repeat), skips strictly future KV tiles (i_kv * B_kv > (i_q + 1) * B_q - 1) via jax.lax.cond, and unpacks [B, B_q, B_kv] tile masks on the fly in VMEM from 8 uint32 words per KV tile (ORing current-block self-attention and ANDing causal, 2D padding, and packed segment_ids document masks).
  • FlashAttention-2 LSE @jax.custom_vjp backward: Saves only (q, k, v, packed_mask, out, lse) (O(B * S) memory; lse is 3 MB at B=2, S=8192) and recomputes tile probabilities on the fly in backward. Ties stop_gradient integer masks and loop tile indices to dynamic q, k, v inputs via a zero-valued jax.lax.optimization_barrier scalar so JAX's _scan_known_hoisting / _cond_partial_eval under outer jax.lax.scan + remat_policy=custom (save_only_these_names("query_proj")) never hoists or stacks 6D/7D [num_q_tiles, num_kv_tiles, B, B_q, H_kv, G, B_kv] branch residuals (0.00 GiB stacked tile residuals in full-model HLO), and preserves (data, tensor) sharding (h_kv = 1 per TP rank on tp=2) across 6D/5D tile reshapes.
  • Strategy-aware sharding & query-side length_axes jax.shard_map: Resolves batch, heads (tensor + tensor_sequence, with GQA k/v repeat when head_shards > num_kv_heads), and length (context and EP=CP expert) via self.sharding, packs bitmaps via jax.vmap(jax.vmap(...)), and scans local_q_tiles = s_q_loc // b_q per sequence shard with zero query all-gathers.
  • Default & Fallback: Keeps qsa_attention_kernel="dense" as the default config setting and dispatches qsa_attention_kernel="flash" through Tokamax Splash Attention when S <= token_budget and flash_block_sparse_attention when S > token_budget.

FIXES: b/580120934

Tests

  1. 4-CPU Unit & Sharding Tests (tests/unit/m3/qwen3_8_flash_next/modeling_qwen3_8_flash_next_test.py, tests/unit/m3/qwen3_8_rope_precision_test.py):

    • JAX_PLATFORMS=cpu XLA_FLAGS="--xla_force_host_platform_device_count=4" pytest -q tests/unit/m3/qwen3_8_flash_next/modeling_qwen3_8_flash_next_test.py -k "qsa" tests/unit/m3/qwen3_8_rope_precision_test.py: 8 passed, 2 skipped, 13 subtests passed.
    • Verifies forward and backward (nnx.value_and_grad) numerical parity between qsa_attention_kernel="flash" and qsa_attention_kernel="dense" across unpacked, mask_2d, packed_seg, and both_mask_and_seg in both float32 (max_diff <= 1.5e-5) and bfloat16, multi-word 32-bit bitmap packing with duplicate indices and -1 padding, non-multiple-of-compress_ratio tail blocks (S=70, c=4), multi-word fast unpacked path (S=256, B_kv=128), packed 2-document isolation (packed_out == concat(unpacked_doc1, unpacked_doc2) in forward and backward), Qwen3_8QSAIndexer(..., return_token_mask=False) bit-identical selected_block_indices parity with zero O(B * S^2) bool[B, S, S] allocation, 4-device (data=2, tensor=2) mesh sharding (verifying per-shard h_kv = 1 in compiled HLO, with explicit self.skipTest when < 4 devices), and outer jax.lax.scan + jax.checkpoint (save_only_these_names("query_proj"), nothing_saveable, everything_saveable) residual check asserting zero 6D/7D stacked tile buffers and zero bool[B, S, S] indexer masks.
    • Negative controls: Corrupting a non-causal selected block bit in packed_mask or dropping segment_ids across packed documents produces max_abs_diff > 0.05 against the exact output, proving the tests fail if sparse block masking or packed document isolation is broken.
  2. Module-Level Qwen3_8SparseAttention (Indexer + Projections + Attention, Forward + Backward) tpu7x-8 AOT HBM ([compiled AOT tpu7x-8], data=4, tensor=2, per-device B=1, global B=4):

Sequence Length S qsa_attention_kernel Status Temp HBM ([compiled AOT tpu7x-8]) Peak HBM ([compiled AOT tpu7x-8]) Temp HBM Savings vs. dense
S = 4096 dense PASS 2.773 GiB 3.038 GiB —
flash PASS 0.712 GiB 0.976 GiB -2.061 GiB (-74.3%)
S = 8192 dense PASS 10.080 GiB 10.384 GiB —
flash PASS 1.462 GiB 1.766 GiB -8.618 GiB (-85.5%)
S = 16384 dense PASS 38.323 GiB 38.705 GiB —
flash PASS 2.824 GiB 3.206 GiB -35.499 GiB (-92.6%, 12.1x peak reduction)
  1. Full-Model 4-Layer qwen3.8-flash-next train_compile.py AOT on tpu7x-8 (ici_fsdp_parallelism=4, ici_tensor_parallelism=2, remat_policy=custom, decoder_layer_input=offload) ([compiled AOT tpu7x-8]):
Sequence Length S qsa_attention_kernel 6D/7D Stacked Branch Residuals Dense [2, 24, S, S] Buffers Compiler Peak memory HLO Total bytes (preallocated-temp) TRAIN-KERNEL DEVICE PEAK (Temp) Generated Code Size
S = 4096 dense 0 (0.00 GiB) 4 x 3.00 = 12.00 GiB 62.28 GiB 83.30 GiB (55.44 GiB) 112.21 GiB (84.36 GiB) 339.9 MiB
flash (06c7daec4) 0 (0.00 GiB) 0 (0.00 GiB) 62.54 GiB 83.95 GiB (56.09 GiB) 113.67 GiB (85.81 GiB) 187.4 MiB (-44.9%)
S = 8192 dense 0 (0.00 GiB) 2 x 12.00 + 2 x 6.00 = 36.00 GiB 73.73 GiB 79.31 GiB (51.45 GiB) 88.60 GiB (60.75 GiB) 509.9 MiB
flash (06c7daec4) 0 (0.00 GiB) 0 (0.00 GiB) 74.51 GiB 76.20 GiB (-3.11 GiB) 82.16 GiB (54.31 GiB, -6.44 GiB / -10.6% Temp) 187.4 MiB (-63.2%)

(In after_optimizations-live-range.txt, the global live-range Peak memory of ~62.3-62.5 GiB at S=4096 and ~73.7-74.5 GiB at S=8192 occurs inside the 512-expert MoE backward pass tgmm_v2 + 25.00 GiB expert weight-gradient all-reduce + 27.86 GiB model/optimizer state, not inside QSA.)

  1. Measured 25-Step tpu7x-8 (2x2x1, bodaborg-tpu7x-nap) End-to-End Training A/B ([measured tpu7x-8]):
    • Cloud Logging share link: https://cloudlogging.app.goo.gl/J5xKk9EAUfcHUoJy7
    • (Note: The 25-step runtime timing below was captured on initial commit f2d94500, which executes the identical inner tiled FlashAttention loop prior to ab11fad28's -26.94 GiB remat_policy=custom branch-residual fix documented in the AOT table above.)
Case (4L qwen3.8-flash-next, tpu7x-8, fsdp=4, tp=2) p50 Step Time ([measured tpu7x-8]) Throughput (TFLOP/s/dev) Speedup vs. dense Step 0 Loss Step 24 Loss Step 24 grad_norm Cloud Logging Share Link
qsa_4L_s4096_dense (run 1 / run 2) 4383.0 / 4382.5 ms 6.563 1.00x 12.545 9.719 0.626928 https://cloudlogging.app.goo.gl/J5xKk9EAUfcHUoJy7
qsa_4L_s4096_flash 1062.0 ms 27.078 4.13x faster 12.545 9.718 0.626921 https://cloudlogging.app.goo.gl/J5xKk9EAUfcHUoJy7
qsa_4L_s8192_dense (run 1 / run 2) 15086.0 / 15085.0 ms 3.825 1.00x 12.546 10.113 0.482197 https://cloudlogging.app.goo.gl/J5xKk9EAUfcHUoJy7
qsa_4L_s8192_flash 1635.5 ms 35.279 9.22x faster 12.546 10.113 0.482203 https://cloudlogging.app.goo.gl/J5xKk9EAUfcHUoJy7
2026-11-12T03:50:21.456238021Z === STAGE_START: qsa_4L_s4096_dense__run2 ===
2026-11-12T03:53:41.139404433Z completed step: 0, seconds: 0.190, TFLOP/s/device: 151.574, Tokens/s/device: 21583.795, total_weights: 32768, loss: 12.545, lm_loss: 12.545, perplexity: 280765.250, moe_lb_loss: 0.000000, diag: raw_grad_norm=9.647469521e-01 grad_norm=9.647473693e-01 param_norm=8.544595947e+02 max_abs_grad=1.840636600e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T03:54:33.736819769Z completed step: 12, seconds: 4.384, TFLOP/s/device: 6.562, Tokens/s/device: 934.365, total_weights: 32768, loss: 10.414, lm_loss: 10.414, perplexity: 33320.648, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.836047173e-01 grad_norm=6.836051345e-01 param_norm=8.542636108e+02 max_abs_grad=5.068879575e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T03:57:07.826761265Z completed step: 24, seconds: 3.966, TFLOP/s/device: 7.253, Tokens/s/device: 1032.860, total_weights: 32768, loss: 9.719, lm_loss: 9.719, perplexity: 16624.637, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.269273162e-01 grad_norm=6.269276142e-01 param_norm=8.542371826e+02 max_abs_grad=5.410716869e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T03:57:11.559791004Z [CASE_RUN_SUMMARY] {"case_id": "qsa_4L_s4096_dense", "run_idx": 2, "case_run_id": "qsa_4L_s4096_dense__run2", "sha8": "f2d94500", "rc": 0, "wall_sec": 410.1, "num_logged_steps": 25, "p50_step_ms": 4383.0, "p50_tflops_per_dev": 6.563, "mfu_vs_1150_pct": 0.57, "peak_hbm_gib": 75.1229, "compiled_hbm_gib": 75.1229, "raw_peak_hbm_gib": 67.0929, "host_arg_hbm_gib": 0.0, "peak_alloc_hbm_gib": 28.1227, "peak_in_use_hbm_gib": 28.1227, "alloc_hbm_gib": 0.0012, "final_step": 24, "final_loss": 9.719, "final_grad_norm": 0.6269276142, "final_raw_grad_norm": 0.6269273162, "step0_loss": 12.545, "step0_grad_norm": 0.9647473693, "step0_raw_grad_norm": 0.9647469521}
2026-11-12T04:01:18.349822981Z completed step: 0, seconds: 0.262, TFLOP/s/device: 109.865, Tokens/s/device: 15644.515, total_weights: 32768, loss: 12.545, lm_loss: 12.545, perplexity: 280767.469, moe_lb_loss: 0.000000, diag: raw_grad_norm=9.647405744e-01 grad_norm=9.647408724e-01 param_norm=8.544595947e+02 max_abs_grad=1.839970937e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T04:01:31.106338703Z completed step: 12, seconds: 1.063, TFLOP/s/device: 27.064, Tokens/s/device: 3853.797, total_weights: 32768, loss: 10.414, lm_loss: 10.414, perplexity: 33328.844, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.836234331e-01 grad_norm=6.836239696e-01 param_norm=8.542636108e+02 max_abs_grad=5.075586494e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:13:14.382340882Z completed step: 24, seconds: 0.633, TFLOP/s/device: 45.440, Tokens/s/device: 6470.580, total_weights: 32768, loss: 9.718, lm_loss: 9.718, perplexity: 16621.100, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.269207001e-01 grad_norm=6.269209981e-01 param_norm=8.542371826e+02 max_abs_grad=5.409580190e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:13:18.440562851Z [CASE_RUN_SUMMARY] {"case_id": "qsa_4L_s4096_flash", "run_idx": 1, "case_run_id": "qsa_4L_s4096_flash__run1", "sha8": "f2d94500", "rc": 0, "wall_sec": 368.07, "num_logged_steps": 25, "p50_step_ms": 1062.0, "p50_tflops_per_dev": 27.078, "mfu_vs_1150_pct": 2.35, "peak_hbm_gib": 84.8408, "compiled_hbm_gib": 84.8408, "raw_peak_hbm_gib": 67.089, "host_arg_hbm_gib": 0.0, "peak_alloc_hbm_gib": 28.1281, "peak_in_use_hbm_gib": 28.1281, "alloc_hbm_gib": 0.0012, "final_step": 24, "final_loss": 9.718, "final_grad_norm": 0.6269209981, "final_raw_grad_norm": 0.6269207001, "step0_loss": 12.545, "step0_grad_norm": 0.9647408724, "step0_raw_grad_norm": 0.9647405744}
2026-11-12T05:27:40.536263237Z === STAGE_START: qsa_4L_s8192_dense__run2 ===
2026-11-12T05:30:55.514428036Z completed step: 0, seconds: 0.203, TFLOP/s/device: 283.878, Tokens/s/device: 40306.233, total_weights: 65536, loss: 12.546, lm_loss: 12.546, perplexity: 280961.281, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.807751060e-01 grad_norm=6.807755232e-01 param_norm=8.544595947e+02 max_abs_grad=1.309303101e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:33:56.539964655Z completed step: 12, seconds: 15.085, TFLOP/s/device: 3.825, Tokens/s/device: 543.066, total_weights: 65536, loss: 10.712, lm_loss: 10.712, perplexity: 44871.277, moe_lb_loss: 0.000000, diag: raw_grad_norm=5.169830918e-01 grad_norm=5.169832706e-01 param_norm=8.542973022e+02 max_abs_grad=4.021274857e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:39:41.532005350Z completed step: 24, seconds: 14.498, TFLOP/s/device: 3.980, Tokens/s/device: 565.047, total_weights: 65536, loss: 10.113, lm_loss: 10.113, perplexity: 24669.711, moe_lb_loss: 0.000000, diag: raw_grad_norm=4.821972251e-01 grad_norm=4.821974337e-01 param_norm=8.542850342e+02 max_abs_grad=4.489537794e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:39:45.228812601Z [CASE_RUN_SUMMARY] {"case_id": "qsa_4L_s8192_dense", "run_idx": 2, "case_run_id": "qsa_4L_s8192_dense__run2", "sha8": "f2d94500", "rc": 0, "wall_sec": 724.69, "num_logged_steps": 25, "p50_step_ms": 15085.0, "p50_tflops_per_dev": 3.825, "mfu_vs_1150_pct": 0.33, "peak_hbm_gib": 87.2934, "compiled_hbm_gib": 87.2934, "raw_peak_hbm_gib": 78.0762, "host_arg_hbm_gib": 0.0, "peak_alloc_hbm_gib": 28.1343, "peak_in_use_hbm_gib": 28.1343, "alloc_hbm_gib": 0.0012, "final_step": 24, "final_loss": 10.113, "final_grad_norm": 0.4821974337, "final_raw_grad_norm": 0.4821972251, "step0_loss": 12.546, "step0_grad_norm": 0.6807755232, "step0_raw_grad_norm": 0.680775106}
2026-11-12T05:39:45.229167934Z === STAGE_START: qsa_4L_s8192_flash__run1 ===
2026-11-12T05:45:21.366373533Z completed step: 0, seconds: 0.260, TFLOP/s/device: 222.248, Tokens/s/device: 31555.754, total_weights: 65536, loss: 12.546, lm_loss: 12.546, perplexity: 280966.125, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.807749271e-01 grad_norm=6.807753444e-01 param_norm=8.544595947e+02 max_abs_grad=1.309968531e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:45:40.994759873Z completed step: 12, seconds: 1.636, TFLOP/s/device: 35.258, Tokens/s/device: 5006.074, total_weights: 65536, loss: 10.712, lm_loss: 10.712, perplexity: 44872.227, moe_lb_loss: 0.000000, diag: raw_grad_norm=5.169836283e-01 grad_norm=5.169838071e-01 param_norm=8.542972412e+02 max_abs_grad=4.025802016e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:48:34.495506338Z completed step: 24, seconds: 1.022, TFLOP/s/device: 56.448, Tokens/s/device: 8014.769, total_weights: 65536, loss: 10.113, lm_loss: 10.113, perplexity: 24669.629, moe_lb_loss: 0.000000, diag: raw_grad_norm=4.822026491e-01 grad_norm=4.822028577e-01 param_norm=8.542850342e+02 max_abs_grad=4.495411180e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T05:48:39.176136766Z [CASE_RUN_SUMMARY] {"case_id": "qsa_4L_s8192_flash", "run_idx": 1, "case_run_id": "qsa_4L_s8192_flash__run1", "sha8": "f2d94500", "rc": 0, "wall_sec": 533.95, "num_logged_steps": 25, "p50_step_ms": 1635.5, "p50_tflops_per_dev": 35.279, "mfu_vs_1150_pct": 3.07, "peak_hbm_gib": 91.7383, "compiled_hbm_gib": 91.7383, "raw_peak_hbm_gib": 79.1489, "host_arg_hbm_gib": 0.0, "peak_alloc_hbm_gib": 28.1437, "peak_in_use_hbm_gib": 28.1437, "alloc_hbm_gib": 0.0012, "final_step": 24, "final_loss": 10.113, "final_grad_norm": 0.4822028577, "final_raw_grad_norm": 0.4822026491, "step0_loss": 12.546, "step0_grad_norm": 0.6807753444, "step0_raw_grad_norm": 0.6807749271}
2026-11-12T05:48:39.176185676Z === STAGE_END: qsa_4L_s8192_flash__run1 ===
  1. Post-Fix (ab11fad28) Native FSDP=2, EP=4 tpu7x-8 (2x2x1, bodaborg-tpu7x-nap) Measured Results ([measured tpu7x-8]):
    • Cloud Logging share link: https://cloudlogging.app.goo.gl/PRQ53vtJpWa4F9Dk8
    • Pod: m3-qsa-ab11-2x2x1-bench-0-0-5nncp (measured with native FSDP=2, EP=4 sharding without sharding_tolerance override):
      • S = 4096: 2397.75 ms -> 732.00 ms (3.28x faster, throughput 12.00 -> 39.26 TFLOP/s/dev, runtime compiled HBM 43.62 -> 43.44 GiB [-0.18 GiB], live allocated peak 16.49 -> 16.50 GiB, Step 14 loss 10.204 -> 10.204 [0.000 diff], Step 0 raw_grad_norm relative diff 4.60e-6).
      • S = 8192: 7979.75 ms -> 1261.50 ms (6.33x faster, throughput 7.23 -> 45.85 TFLOP/s/dev, runtime compiled HBM 63.25 -> 63.47 GiB [0.00 GiB raw peak delta, 56.32 -> 56.32 GiB], live allocated peak 16.54 -> 16.55 GiB, Step 14 loss 10.530 -> 10.530 [0.000 diff], Step 0 raw_grad_norm relative diff 8.09e-6).
Case (4L qwen3.8-flash-next, tpu7x-8, native fsdp=2, ep=4) p50 Step Time ([measured tpu7x-8]) Throughput (TFLOP/s/dev) Speedup vs. dense Step 14 Loss Runtime Compiled HBM Live Allocated Peak Evidence
S = 4096 dense (2 runs) 2397.75 ms (2398.0 / 2397.5) 12.00 1.00x 10.204 43.62 GiB (raw 37.74) 16.49 GiB logs
S = 4096 flash (1 run) 732.00 ms 39.26 3.28x 10.204 (0.000 diff) 43.44 GiB (-0.18 GiB) 16.50 GiB logs
S = 8192 dense (2 runs) 7979.75 ms (7979.0 / 7980.5) 7.23 1.00x 10.530 63.25 GiB (raw 56.32) 16.54 GiB logs
S = 8192 flash (steps 0..17) 1261.50 ms 45.85 6.33x 10.530 (0.000 diff) 63.47 GiB (0.00 GiB raw delta) 16.55 GiB logs
2026-11-12T06:56:35.558115792Z [TPU_COMPILED_HBM] case_run=qsa_ep4_4L_s4096_dense__run1 total_gib=43.6219 temp_gib=27.3504 arg_gib=16.2715 raw_peak_gib=37.7362 host_arg_gib=0.0000
2026-11-12T06:57:12.867840449Z completed step: 14, seconds: 2.398, TFLOP/s/device: 11.993, Tokens/s/device: 1707.745, total_weights: 32768, loss: 10.204, lm_loss: 10.204, perplexity: 27003.766, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.632491350e-01 grad_norm=6.632494926e-01 param_norm=8.542951050e+02 max_abs_grad=5.131395999e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T07:03:56.077269134Z [TPU_COMPILED_HBM] case_run=qsa_ep4_4L_s4096_flash__run1 total_gib=43.4439 temp_gib=27.1723 arg_gib=16.2715 raw_peak_gib=37.7362 host_arg_gib=0.0000
2026-11-12T07:04:08.554311865Z completed step: 14, seconds: 0.733, TFLOP/s/device: 39.260, Tokens/s/device: 5590.580, total_weights: 32768, loss: 10.204, lm_loss: 10.204, perplexity: 26999.695, moe_lb_loss: 0.000000, diag: raw_grad_norm=6.632692218e-01 grad_norm=6.632696390e-01 param_norm=8.542950439e+02 max_abs_grad=5.170019343e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T07:07:47.287095703Z [TPU_COMPILED_HBM] case_run=qsa_ep4_4L_s8192_dense__run1 total_gib=63.2537 temp_gib=46.9821 arg_gib=16.2716 raw_peak_gib=56.3228 host_arg_gib=0.0000
2026-11-12T07:09:48.453324796Z completed step: 14, seconds: 7.980, TFLOP/s/device: 7.230, Tokens/s/device: 1026.520, total_weights: 65536, loss: 10.530, lm_loss: 10.530, perplexity: 37410.445, moe_lb_loss: 0.000000, diag: raw_grad_norm=5.034753084e-01 grad_norm=5.034757257e-01 param_norm=8.543319702e+02 max_abs_grad=4.638320301e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T07:20:48.392037093Z [TPU_COMPILED_HBM] case_run=qsa_ep4_4L_s8192_flash__run1 total_gib=63.4726 temp_gib=47.2010 arg_gib=16.2716 raw_peak_gib=56.3228 host_arg_gib=0.0000
2026-11-12T07:21:08.838201552Z completed step: 14, seconds: 1.263, TFLOP/s/device: 45.696, Tokens/s/device: 6488.163, total_weights: 65536, loss: 10.530, lm_loss: 10.530, perplexity: 37408.395, moe_lb_loss: 0.000000, diag: raw_grad_norm=5.034763813e-01 grad_norm=5.034767985e-01 param_norm=8.543319092e+02 max_abs_grad=4.627875052e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00
2026-11-12T07:21:12.621847145Z completed step: 17, seconds: 1.258, TFLOP/s/device: 45.846, Tokens/s/device: 6509.476, total_weights: 65536, loss: 10.328, lm_loss: 10.328, perplexity: 30579.602, moe_lb_loss: 0.000000, diag: raw_grad_norm=4.919891655e-01 grad_norm=4.919895232e-01 param_norm=8.543276367e+02 max_abs_grad=4.814887419e-03 moe_bias_checksum=0.000000000e+00 moe_bias_update_nonzero=0.000000000e+00 moe_overflow=0.000000000e+00

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a memory-efficient tiled block-sparse FlashAttention kernel for Qwen3.8 Sparse Attention. By replacing dense mask materialization with a compact 32-bit packed bitmap and implementing a tiled online-softmax approach, the changes allow for training on much longer sequences while maintaining numerical parity with the existing dense implementation. The solution includes robust sharding support and custom VJP backward passes to optimize memory footprint during training.

Highlights

  • Tiled Block-Sparse FlashAttention: Implemented a new tiled block-sparse FlashAttention path for Qwen3.8 Sparse Attention (QSA) to handle sequence lengths exceeding the token budget, significantly reducing HBM usage.
  • 32-bit Packed Block Bitmap: Introduced a carry-free 32-bit packed block bitmap (pack_qsa_block_mask_32bit) to efficiently represent selected block indices without dense mask materialization.
  • Memory Optimization: Utilized JAX custom VJP for FlashAttention-2 LSE backward pass, enabling O(B*S) memory complexity and preventing the stacking of high-dimensional branch residuals.
  • Performance Gains: Achieved up to 9.22x speedup and significant peak HBM reduction (up to 12.1x at S=16384) compared to the dense attention implementation.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.95918% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
.../qwen3_8_flash_next/modeling_qwen3_8_flash_next.py 97.95% 3 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: High-quality implementation; FlashAttention-2 custom VJP and 32-bit packed bitmap mask cleanly resolve the QSA OOM while preserving numerical parity and sharding.

Top issues:

  1. [Perf] _compute_qsa_doc_start eliminates $O(S^2)$ masks in attention, but Qwen3_8QSAIndexer.__call__ still evaluates same_doc = (segment_ids[:, :, None] == segment_ids[:, None, :]) to compute doc_start even when return_token_mask=False. → In Qwen3_8QSAIndexer, reuse _compute_qsa_doc_start(segment_ids) when not return_token_mask to avoid $O(B \cdot S^2)$ boolean allocations during multi-doc indexing (see inline comment on modeling_qwen3_8_flash_next.py).
  2. [Tests] Section 4 of test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls guards 4-device mesh sharding with if len(jax.devices()) >= 4:, silently passing without execution on single-device environments. → Split the multi-device check into a dedicated test method with if len(jax.devices()) < 4: self.skipTest(...) matching lines 1414 and 2018 (see inline comment on modeling_qwen3_8_flash_next_test.py).
Inferred spec & validation
  • Sources: PR #6283 description; code before/after; internal ticket b/580120934 (not readable here).
  • Problem: For sequence lengths exceeding the token budget ($S &gt; 2048$), Qwen3_8SparseAttention previously materialized dense $O(S^2)$ attention matrices in HBM ($36\text{ GiB}$ at $S=8192$, OOM at $97.5\text{ GiB}$ at $S=16384$), while qsa_attention_kernel="flash" rejected $S &gt; 2048$ with a ValueError.
Spec item Implementation Validating test
R1 (stated): Implement tiled block-sparse FlashAttention (flash_block_sparse_attention) for $S &gt; \text{token_budget}$ ✅ met modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation ✅
R2 (stated): Compact 32-bit uint32 packed block bitmap via deduplicated scatter-add without materializing booleans ✅ met modeling_qwen3_8_flash_next_test.py::test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls (section 1) ✅
R3 (stated): Tiled online-softmax forward with causal tile skipping, GQA head broadcasting, and VMEM mask unpacking ✅ met modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation (unpacked/mask_2d/packed_seg/both) ✅
R4 (stated): FlashAttention-2 custom VJP backward recomputing tile probabilities with compiler optimization barrier ✅ met modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation (value_and_grad), section 5 (jaxpr check) ✅
R5 (stated): Multi-document isolation (segment_ids), 2D attention_mask padding, and non-divisible tail block handling ✅ met modeling_qwen3_8_flash_next_test.py::test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls (sections 2, 3) ✅
C1 (stated): Backward compatibility; keep qsa_attention_kernel="dense" as default in types.py ✅ preserved modeling_qwen3_8_flash_next_test.py::test_kernel_path_flags_match_defaults_end_to_end ✅
C2 (stated): Forward and backward numerical parity against dense reference in float32 and bfloat16 ✅ preserved modeling_qwen3_8_flash_next_test.py::test_flash_attention_kernel_parity_and_validation ✅
C3 (stated): Preservation of (data, tensor) sharding across tile transformations without communication inside kernel ✅ preserved modeling_qwen3_8_flash_next_test.py::test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls (section 4) ✅

self.assertGreater(float(jnp.max(jnp.abs(out_leaked[:, len1:, :] - out_cat[:, len1:, :]))), 0.05)

# 4. 4-device CPU sharded parity (data=2, tensor=2) for s > token_budget.
if len(jax.devices()) >= 4:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Guarding multi-device verification with if len(jax.devices()) >= 4: causes this sub-test to pass silently without executing on standard single-device CPU test runs. Consider splitting section 4 into a dedicated test method test_qsa_flash_block_sparse_4device_sharded_parity with if len(jax.devices()) < 4: self.skipTest(...), following the pattern in lines 1414 and 2018 so that test skips are explicitly reported by unittest.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: extracted section 4 into a dedicated test_qsa_flash_block_sparse_4device_sharded_parity test method that calls self.skipTest("Requires >= 4 JAX devices") when len(jax.devices()) < 4.

return token_mask, selected_block_indices


def _compute_qsa_doc_start(segment_ids: jax.Array) -> jax.Array:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 _compute_qsa_doc_start computes document start offsets in O(B * S log S) without O(S^2) masks and is used in flash_block_sparse_attention, but Qwen3_8QSAIndexer.__call__ (line 660) still evaluates same_doc = (segment_ids[:, :, None] == segment_ids[:, None, :]) & (segment_ids[:, :, None] != 0) to derive doc_start even when return_token_mask=False. For long multi-document sequences, reusing _compute_qsa_doc_start in Qwen3_8QSAIndexer when not return_token_mask will eliminate the remaining O(B * S^2) boolean allocation in the indexer.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: moved _compute_qsa_doc_start above Qwen3_8QSAIndexer and used it (with a fast jnp.zeros path when unpacked) to compute doc_start in Qwen3_8QSAIndexer.__call__, deferring the O(B * S^2) same_doc allocation until return_token_mask=True.

@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-qwen38-qsa-block-sparse branch from ab11fad to 06c7dae Compare November 12, 2026 11:12
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

@AusarYao28 AusarYao28 removed skip-ci-tests Add this label to the PR to skip running CI tests gemini-review labels Nov 12, 2026
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-qwen38-qsa-block-sparse branch from 72a612e to 06b61f1 Compare November 13, 2026 10:13
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found. The block-sparse FlashAttention implementation elegantly eliminates the O(B*S^2) mask materialization, handles tail blocks and multi-document packing accurately, and correctly prevents loop-carried residuals in the custom VJP.

Inferred spec & validation
  • Sources: PR description, b/580120934 (inferred from description context as Buganizer is internal), code before/after.
  • Problem: When sequence length > 2048, Qwen3_8SparseAttention materialized a dense [B, S, S] boolean mask and dense logits, causing massive HBM usage and OOM on TPU at S=16384.
Spec item Implementation Validating test
R1 (stated): Pack block mask into 32-bit uint32 bitmap. ✅ met test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅
R2 (stated): Tiled online-softmax forward skipping causal future tiles. ✅ met test_flash_attention_kernel_parity_and_validation ✅
R3 (stated): Custom VJP backward saving only required activations to avoid O(B*S^2) memory. ✅ met test_flash_attention_kernel_parity_and_validation ✅
R4 (stated): Prevent 6D/7D branch residuals from being stacked by outer scan remat. ✅ met test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅
C1 (stated): Fallback to Tokamax Splash Attention when S <= token_budget and keep dense as default. ✅ preserved test_flash_attention_kernel_parity_and_validation ✅
C2 (inferred): Retain precision and numerical equivalence with dense fallback path in bf16 and fp32. ✅ preserved test_flash_attention_kernel_parity_and_validation ✅

@tissue3

tissue3 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Hi @AusarYao28, thanks for the heads-up!

1. Rebase of #6274 onto #6205 (e00130aaa) & 4D gate_up_proj Verification

I rebased #6274 onto the latest yaoyuchen/m3-host-offload-hbm-telemetry (e00130aaa, including #6164's 4D gate_up_proj and #6152's 8-axis MESH_AXIS_NAMES with "tensor_sequence").

4D Qwen3_8Experts.gate_up_proj (E, D, 2, M) works cleanly with #6274:

  • Unscanned gate_up_proj (("experts", "expert_embed", None, "expert_mlp")) resolves to P("expert", ("fsdp", "fsdp_transpose"), None, ("tensor", "tensor_sequence")).
  • Scanned cycles.mlp.experts.gate_up_proj resolves to P(None, "expert", ("fsdp", "fsdp_transpose"), None, ("tensor", "tensor_sequence")).
  • All 21/21 unit tests in tests/unit/m3/qwen3_8_flash_next/qwen3_8_sharding_test.py pass across 4 devices (0 skipped, 0 failed), verifying shard shapes (8, 128, 2, 32) under tensor=2, (4, 128, 2, 64) under expert=2, and (6, 64, 2, 32) under indivisible expert=4 (num_experts=6 -> tp_axes=("expert",)), plus Checks 2–8 of tests/unit/m3/conformance/conformance_test.py (including Check 7 weight sharding coverage and Check 8 no 4D/5D expert weight all-gather).
  • Two small notes on feat(m3): add pinned-host offload, runtime TPU HBM telemetry, Qwen3.8 fp64 RoPE, and Qwen3-VL FLOPs #6205's _run_qwen3_8_sublayer:
    1. Since feat(m3): add pinned-host offload, runtime TPU HBM telemetry, Qwen3.8 fp64 RoPE, and Qwen3-VL FLOPs #6205 wraps the unscanned layer body in _run_qwen3_8_sublayer (jax.checkpoint(..., prevent_cse=True)), _gdn_chunk_transfer(xs, state_shape) at modeling_qwen3_8_flash_next.py:483 needed @functools.partial(jax.checkpoint, static_argnums=(1,)) (included in Add m3 sharding annotations for Qwen3.8-Flash-Next #6274) so state_shape: tuple[int, int, int, int] stays concrete inside jnp.eye(chunk_size) under nested jax.checkpoint + jax.value_and_grad.
    2. In conformance_test Check 1 (qwen3.8-flash-next[scan=False,remat=full]), prevent_cse=True inside the unscanned _run_qwen3_8_sublayer inserts optimization barriers that change XLA fp32 fusion boundaries relative to remat=none (2.42e-5 > 1e-5 Frobenius diff under --xla_backend_optimization_level=0); setting prevent_cse=False on the unscanned path (matching glm5_3_flash / qwen3, since prevent_cse is only needed inside jax.lax.scan) avoids that forward/grad fusion drift.

2. First-Pass Review of #6283 (Tiled Block-Sparse FlashAttention for QSA)

The algorithm and custom VJP design in #6283 look great:

  • Replacing the $O(B \cdot S^2)$ dense causal/document mask in the indexer with _compute_qsa_doc_start ($O(B \cdot S \log S)$ sort + associative scan) eliminates the quadratic mask materialization.
  • Packing selected_block_indices into uint32[B, S, num_words] via pack_qsa_block_mask_32bit and using @jax.custom_vjp + _unk_zero = jax.lax.optimization_barrier(...) cleanly prevents scan_layers / remat from hoisting tile indices or stacking [num_q_tiles, num_kv_tiles, ...] branch residuals.

Two sharding items to flag in flash_block_sparse_attention (src/maxtext/m3/models/qwen3_8_flash_next/modeling_qwen3_8_flash_next.py:155-245) for when #6283 and #6274 merge:

  1. get_batch_mesh_axes() called without self.sharding & hardcoded "tensor" (lines 156–161):

    • Line 156 calls batch_axes = get_batch_mesh_axes() with no sharding argument (defaulting to expert_context_parallel=False). Under expert_context_parallel=True (EP = CP), b_spec will include "expert", colliding with the sequence axis.
    • Lines 159–161 hardcode tp_size = int(mesh.shape.get("tensor", 1)) and tp_spec = "tensor" instead of get_heads_mesh_axes(self.sharding) (which also includes "tensor_sequence").
    • Suggestion: Pass sharding: Sharding = DefaultSharding() into flash_block_sparse_attention from Qwen3_8SparseAttention._block_sparse_attention and compute b_spec = get_batch_mesh_axes(sharding) and tp_spec = get_heads_mesh_axes(sharding), matching _flash_attention (modeling_qwen3_8_flash_next.py:1255-1258).
  2. Preserving query-side sequence sharding (length_axes) across tiled Q scan (lines 168–173, 235–243):

    • Lines 168–173 constrain q, packed_mask, seg_q, and local_q with None on axis 1 (the sequence axis). When context > 1 or expert_context_parallel=True, q and selected_block_indices enter sharded along length_axes (P(batch, length, ...)). Constraining axis 1 to None forces an All-Gather / All-to-All of the query-side tensors off length_axes, and every sequence shard redundantly scans all num_q_tiles queries. (Replicating k and v on axis 1 is expected for causal attention, matching _flash_attention's kv_spec.)
    • Suggestion: Note that simply changing None to length_axes in _constrain wouldn't avoid the All-Gather because q_tiled moves num_q_tiles to axis 0 and jax.lax.scans over it (lines 235–243, 329–339). Wrapping the tiled scan in jax.shard_map over length_axes (analogous to Splash's q_seq_shards) lets each sequence shard scan only its local_q_tiles = num_q_tiles // q_shards, offset global q_tile_idx by lax.axis_index(length_axes) * local_q_tiles for the causal/bitmap bounds (kv_end <= q_start / kv_start > q_end), and keep k, v replicated along length_axes.

3. Quick Coordination Note on src/maxtext/m3/core/moe.py

In #6274, I made a small backwards-compatible addition to MoEExecutor (length: MeshAxes = None, defaulting to None so GLM-5.3, DeepSeek-V3.2, and Granite are untouched):

  • 3D (B, S, D) activation sharding through MoE: When Qwen3_8MoEBlock runs under context > 1 or expert_context_parallel=True, hidden_states enters sharded as P(batch, length, embed). Passing length to MoEExecutor keeps the router / ragged_dot / _sharded_grouped (gmm_v2) / _ep_ shard_map entry spec on (batch, length, ...) so Qwen3_8MoEBlock stays 3D (B, S, D) across both the shared expert and routed experts without flattening (B, S) -> (B*S) or putting "expert" on both B and S.
  • Follow-up note for tpu7x multi-axis EP (len(ep_axes) > 1): Currently moe.py:856 raises NotImplementedError("Expert sharding supports at most one mesh axis") when len(ep_axes) > 1. On Zebrafish (zf), each chip is a single JAX device so 1D "expert" is sufficient, whereas on tpu7x (dual-TensorCore per chip) a 2D expert axis tuple (e.g. ("context", "expert")) hits this check. Since lax.ragged_all_to_all and lax.all_gather natively accept axis_name=ep_axes as a tuple, linearizing ep_idx the same way tp_idx is linearized at moe.py:907 (~15 lines) can lift this whenever we want multi-axis EP on tpu7x.

@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-host-offload-hbm-telemetry branch from e00130a to a6b500e Compare November 13, 2026 20:07
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-qwen38-qsa-block-sparse branch from 06b61f1 to 7828327 Compare November 13, 2026 20:08
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-host-offload-hbm-telemetry branch from a6b500e to 8d7500a Compare November 14, 2026 00:09
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-qwen38-qsa-block-sparse branch from 7828327 to 7fea1aa Compare November 14, 2026 00:10
@AusarYao28

Copy link
Copy Markdown
Collaborator Author

Thanks so much for verifying 4D gate_up_proj against #6274 and for the detailed review @tissue3! Folded all of your items into the stack:

  1. Unscanned _run_qwen3_8_sublayer prevent_cse=False (Add m3 rematerialization, hybrid loop barriers, and activation savepoints #6249 c6f1f9fc80 -> feat(m3): add pinned-host offload, runtime TPU HBM telemetry, Qwen3.8 fp64 RoPE, and Qwen3-VL FLOPs #6205 8d7500a826):

    • Changed prevent_cse=True -> prevent_cse=False on unscanned _run_qwen3_8_sublayer (keeping prevent_cse=True inside apply_scanned_cycles _eval_slot) so unscanned remat_policy="full" avoids the XLA fp32 fusion boundary drift in Conformance Check 1. Verified < 1e-5 forward/grad drift on both CPU and live 4-chip TPU v5p (test_qwen3_8_unscanned_sublayer_uses_prevent_cse_false_and_matches_no_remat). (Note: yaoyuchen/m3-host-offload-hbm-telemetry is now at 8d7500a826 with that fix included; #6274 already carries @functools.partial(jax.checkpoint, static_argnums=(1,)) on _gdn_chunk_transfer.)
  2. Both sharding items in flash_block_sparse_attention (Add tiled block-sparse FlashAttention with 32-bit packed bitmap mask for m3 Qwen3.8 QSA #6283 7fea1aaacc):

    • self.sharding axis resolution: Qwen3_8SparseAttention.__init__ now stores self.sharding = sharding or DefaultSharding(), passes sharding=self.sharding into flash_block_sparse_attention, and resolves batch_axes, head_axes (covering both "tensor" and "tensor_sequence", plus GQA k, v repetition when head_shards > num_kv_heads), and length_axes ("context" and "expert" under expert_context_parallel=True) via get_batch_mesh_axes / get_heads_mesh_axes / get_length_mesh_axes matching #6274's exact helper signatures.
    • Query-side length_axes shard_map (zero query All-Gather): Rewrote pack_qsa_block_mask_32bit with jax.vmap(jax.vmap(...)) (avoiding the 3D global (b_grid, s_grid) scatter-add that triggered a cross-length_axes all-reduce) and wrapped _kernel in jax.shard_map over (b_spec, length_spec, tp_spec) when length_axes is active and s_pad % (q_seq_shards * b_q) == 0. Each sequence shard scans only local_q_tiles = s_q_loc // b_q (q_tile_indices offset by q_shard_idx * local_q_tiles) while keeping q and packed_mask sharded on length_axes (0 "all-gather(" in compiled HLO) and k, v replicated along length_axes. Verified in test_qsa_flash_block_sparse_sequence_sharding_and_custom_strategy on 4-CPU + live 4-chip TPU v5p and on 4L qwen3.8-flash-next tpu7x-8 AOT train_compile (context=2, tensor=4, context_parallel_load_balance=False).
  3. MoEExecutor length & 2D ep_axes on tpu7x:

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found. The implementation of the tiled block-sparse FlashAttention kernel is mathematically sound, highly optimized, and exceptionally well-tested.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Qwen3_8SparseAttention materialized dense masks and attention logits in HBM when sequence length exceeded the token budget, leading to OOMs and poor MXU utilization due to independent KV gather.
Spec item Implementation Validating test
R1 (stated): Implement a tiled block-sparse FlashAttention kernel for s > token_budget (qsa_attention_kernel="flash"). ✅ test_flash_attention_kernel_parity_and_validation ✅
R2 (stated): Pack the per-query selected_block_indices into a carry-free 32-bit uint32 bitmap. ✅ test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅
R3 (stated): Preserve memory via FlashAttention-2 LSE @jax.custom_vjp that saves O(B * S) memory. ✅ test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅
R4 (stated): Resolve query-side length_axes via jax.shard_map and handle GQA tensor head sharding. ✅ test_qsa_flash_block_sparse_sequence_sharding_and_custom_strategy ✅
C1 (stated): Keep qsa_attention_kernel="dense" as the default configuration. ✅ preserved modeling_qwen3_8_flash_next_test.py ✅
C2 (stated): Fallback to Tokamax Splash Attention when s <= token_budget under the flash kernel. ✅ preserved test_flash_attention_kernel_parity_and_validation ✅

@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-host-offload-hbm-telemetry branch from 8d7500a to 9f3137a Compare November 14, 2026 06:54
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-qwen38-qsa-block-sparse branch from 7fea1aa to 3957c4b Compare November 14, 2026 06:55
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Dense QSA attention materialize [B, S, S] masks and [B, H_q, S, S] FP32 attention tensors causing OOM for sequences > 2048, with poor MXU utilization during local KV gathering.
Spec item Implementation Validating test
R1 (stated): Implement block-sparse FlashAttention path when S > index_select_budget. ✅ test_flash_attention_kernel_parity_and_validation ✅
R2 (stated): Pack QSA selected block masks into a 32-bit uint32 carry-free bitmap. ✅ test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅
R3 (stated): LSE-based custom_vjp to save O(B * S) memory and zero 6D/7D branch residuals. ✅ test_qsa_flash_block_sparse_tail_packing_sharding_and_negative_controls ✅
R4 (stated): Preserve data, tensor, and context sharding without Q all-gathers. ✅ test_qsa_flash_block_sparse_sequence_sharding_and_custom_strategy ✅
C1 (stated): Maintain backward compatibility with qsa_attention_kernel="dense" as default. ✅ preserved test_flash_attention_kernel_parity_and_validation ✅

@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-host-offload-hbm-telemetry branch from 9f3137a to b9a5815 Compare November 17, 2026 03:08
@AusarYao28
AusarYao28 force-pushed the yaoyuchen/m3-qwen38-qsa-block-sparse branch from 3957c4b to 69058e7 Compare November 17, 2026 03:09
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @AusarYao28, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @AusarYao28, but I was unable to process your request. Please see the logs for more details.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants