Skip to content

Sub-channel (1 × 128 and 1 × 512) quantized matrix multiplication applies dequantization scales per K-block inside the contracting reduction loop, which in plain JAX requires 3D batched dots or unrolled HBM passes. Even for full-row (1 × K) scaling, fusing scale + cast into the VMEM matmul tile loop avoids an extra HBM round-trip without relying on compiler downcast-convert fusion. - #166

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1023312592

Conversation

@copybara-service

Copy link
Copy Markdown

Sub-channel (1 × 128 and 1 × 512) quantized matrix multiplication applies dequantization scales per K-block inside the contracting reduction loop, which in plain JAX requires 3D batched dots or unrolled HBM passes. Even for full-row (1 × K) scaling, fusing scale + cast into the VMEM matmul tile loop avoids an extra HBM round-trip without relying on compiler downcast-convert fusion.

Introduce kernels/quantized_matmul.py to support the upcoming custom-kernel execution mode in the gemm_quantized microbenchmark:

  • Add default_quantized_matmul_blocks() to derive lane-aligned (block_m, block_k, block_n, n_lane_multiplier) tiles within the 64 MiB scoped VMEM budget across both on-the-fly (quantize_rhs=True) and weight-offline (quantize_rhs=False) modes.
  • Add quantize_subchannel() to pre-quantize 2D operands into sub-channel blocks (float8_e4m3fn, float8_e5m2, int8, int4) with dynamic or static (0.5 * Q_MAX, scale 0.5) calibration.
  • Add quantized_matmul(), a 3D-grid Pallas TPU kernel supporting both in-tile sub-channel quantization (sc_size <= block_k) and multi-K-tile row-wise quantization (sc_size == K > block_k), homogeneous (FP8 × FP8, INT8 × INT8) and mixed-precision (FP8 × INT4) operands, on-the-fly and weight-offline RHS modes, and optional scale epilogue (dequantize=True/False).

… applies dequantization scales per K-block inside the contracting reduction loop, which in plain JAX requires 3D batched dots or unrolled HBM passes. Even for full-row (`1 × K`) scaling, fusing `scale + cast` into the VMEM matmul tile loop avoids an extra HBM round-trip without relying on compiler downcast-convert fusion.

Introduce `kernels/quantized_matmul.py` to support the upcoming custom-kernel execution mode in the `gemm_quantized` microbenchmark:

*   Add `default_quantized_matmul_blocks()` to derive lane-aligned `(block_m, block_k, block_n, n_lane_multiplier)` tiles within the 64 MiB scoped VMEM budget across both on-the-fly (`quantize_rhs=True`) and weight-offline (`quantize_rhs=False`) modes.
*   Add `quantize_subchannel()` to pre-quantize 2D operands into sub-channel blocks (`float8_e4m3fn`, `float8_e5m2`, `int8`, `int4`) with dynamic or static (`0.5 * Q_MAX`, scale `0.5`) calibration.
*   Add `quantized_matmul()`, a 3D-grid Pallas TPU kernel supporting both in-tile sub-channel quantization (`sc_size <= block_k`) and multi-K-tile row-wise quantization (`sc_size == K > block_k`), homogeneous (`FP8 × FP8`, `INT8 × INT8`) and mixed-precision (`FP8 × INT4`) operands, on-the-fly and weight-offline RHS modes, and optional scale epilogue (`dequantize=True/False`).

PiperOrigin-RevId: 1023312592
@google-cla

google-cla Bot commented Nov 17, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants