Skip to content

Extend the experimental gemm_quantized benchmark in benchmarks/matmul.py to support weight-offline quantization (quantize_rhs: false), modeling inference serving where weight matrices are quantized once offline and only activations are quantized on the fly. - #168

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1020625433

Conversation

@copybara-service

Copy link
Copy Markdown

Extend the experimental gemm_quantized benchmark in benchmarks/matmul.py to support weight-offline quantization (quantize_rhs: false), modeling inference serving where weight matrices are quantized once offline and only activations are quantized on the fly.

  • Add quantize_rhs (bool, default True) to QuantizedGemmParams and QuantizedGemmBenchmark.REPORT_SCHEMA.
  • When quantize_rhs=False, pre-quantize B[K, N] into B_q (quant_dtype) and column scales SF_B (float32) inside generate_inputs() before the timed loop, and quantize only A[M, K] inside the timed run_op step.
  • Count algorithmic HBM bytes for B_q using quant_dtype (m * k * in_bytes + k * n * quant_bytes + m * n * out_bytes) and append _weight_offline to get_run_identifier() when quantize_rhs=False.

…mul.py` to support weight-offline quantization (`quantize_rhs: false`), modeling inference serving where weight matrices are quantized once offline and only activations are quantized on the fly.

*   Add `quantize_rhs` (`bool`, default `True`) to `QuantizedGemmParams` and `QuantizedGemmBenchmark.REPORT_SCHEMA`.
*   When `quantize_rhs=False`, pre-quantize `B[K, N]` into `B_q` (`quant_dtype`) and column scales `SF_B` (`float32`) inside `generate_inputs()` before the timed loop, and quantize only `A[M, K]` inside the timed `run_op` step.
*   Count algorithmic HBM bytes for `B_q` using `quant_dtype` (`m * k * in_bytes + k * n * quant_bytes + m * n * out_bytes`) and append `_weight_offline` to `get_run_identifier()` when `quantize_rhs=False`.

PiperOrigin-RevId: 1020625433
@google-cla

google-cla Bot commented Nov 17, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants