Skip to content

Mixed-precision FP8 × INT4 (W4A8) inference quantizes activations (A) to float8_e4m3fn or float8_e5m2 and weights (B) to int4 while executing on the FP8 matrix unit with float32 accumulation. Previously, gemm_quantized accepted only a single quant_dtype applied identically to both operands. - #164

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1022545780

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Mixed-precision FP8 × INT4 (W4A8) inference quantizes activations (A) to float8_e4m3fn or float8_e5m2 and weights (B) to int4 while executing on the FP8 matrix unit with float32 accumulation. Previously, gemm_quantized accepted only a single quant_dtype applied identically to both operands.

  • Add rhs_quant_dtype to QuantizedGemmParams in benchmarks/matmul.py (defaulting to quant_dtype, including across with_generation_defaults(), and accepting "int4" when quant_dtype is "float8_e4m3fn" or "float8_e5m2").
  • Quantize B to rhs_quant_dtype in QuantizedGemmBenchmark.setup() while keeping float32 accumulation and the quant_dtype MXU roofline ceiling whenever A is FP8.
  • Include rhs_quant_dtype in QuantizedGemmBenchmark.REPORT_SCHEMA and append _rhs_<rhs_quant_dtype> in get_run_identifier() when rhs_quant_dtype differs from quant_dtype.
  • Add TPU + XProf hardware test coverage (QuantizedGemmBenchmarkTest) in tests/test_matmul_tpu.py verifying on-device XProf execution and mixed-precision FP8 × INT4 HLO dtypes.

@google-cla

google-cla Bot commented Nov 16, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@copybara-service
copybara-service Bot force-pushed the test_1022545780 branch 3 times, most recently from f014f3f to da84ee4 Compare November 17, 2026 05:30
… (`A`) to `float8_e4m3fn` or `float8_e5m2` and weights (`B`) to `int4` while executing on the `FP8` matrix unit with `float32` accumulation. Previously, `gemm_quantized` accepted only a single `quant_dtype` applied identically to both operands.

*   Add `rhs_quant_dtype` to `QuantizedGemmParams` in `benchmarks/matmul.py` (defaulting to `quant_dtype`, including across `with_generation_defaults()`, and accepting `"int4"` when `quant_dtype` is `"float8_e4m3fn"` or `"float8_e5m2"`).
*   Quantize `B` to `rhs_quant_dtype` in `QuantizedGemmBenchmark.setup()` while keeping `float32` accumulation and the `quant_dtype` MXU roofline ceiling whenever `A` is `FP8`.
*   Include `rhs_quant_dtype` in `QuantizedGemmBenchmark.REPORT_SCHEMA` and append `_rhs_<rhs_quant_dtype>` in `get_run_identifier()` when `rhs_quant_dtype` differs from `quant_dtype`.
*   Add TPU + XProf hardware test coverage (`QuantizedGemmBenchmarkTest`) in `tests/test_matmul_tpu.py` verifying on-device XProf execution and mixed-precision `FP8 × INT4` HLO dtypes.

PiperOrigin-RevId: 1022545780
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants