Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
| :--- | :--- | :--- | :--- |
| **Compute** | **Matrix Multiplication (GEMM)** | Dense matrix multiplication ($$C = \alpha (A \times B) + \beta C$$) across configurable data types | • Compute Throughput (**TFLOPS**)<br>• Compute Roofline Efficiency (**%**)<br>• Compute Latency (**ms**) |
| **Memory (HBM)** | **HBM Memory Bandwidth** | High Bandwidth Memory STREAM operations (Copy, Scale, Add, Triad, Read-Only, Write-Only) | • Memory Bandwidth (**GB/s**)<br>• Memory Roofline Efficiency (**%**)<br>• Memory Access Latency (**ms**) |
| **Host I/O (PCIe)** | **Host-to-Device (H2D)**<br>**Device-to-Host (D2H)** | Host CPU memory to/from accelerator HBM data transfers | • PCIe Transfer Bandwidth (**GB/s**)<br>• Transfer Latency (**ms**) |
| **Host I/O (PCIe)** | **Host-to-Device (H2D)**<br>**Device-to-Host (D2H)** | Host CPU memory to/from accelerator HBM data transfers | • PCIe Transfer Bandwidth (**GB/s**)<br>• Host Link Roofline Efficiency (**%**)<br>• Transfer Latency (**ms**) |
| **Interconnect (ICI)** | **Device-to-Device (D2D)** | Data transfers between each device pair, over ICI (different chips) or die-to-die (same chip) | • D2D Link Bandwidth (**GB/s**)<br>• Pairwise N × N Device Bandwidth Matrix<br>• Transfer Latency (**ms**) |
| **Collectives** | **All-Gather**<br>**All-Reduce**<br>**All-to-All** | Distributed collective communication across multi-chip topologies | • Collective Bus Bandwidth (**GB/s**)<br>• Collective Step Latency (**ms**) |

Expand Down Expand Up @@ -487,7 +487,7 @@ TPUMS records primary performance metrics across both timing domains:
| **Latency (`ms`)** | `wall_clock_p50_ms`<br>*(+ `p90`, `avg`, `std`)* | `xprof_p50_ms`<br>*(+ `p90`, `avg`, `std`)* | Execution duration across timed iterations (`p50` median, `p90`, mean, and standard deviation). |
| **Compute Throughput (`TFLOPS`)** | `wall_clock_tflops_per_chip`<br>`wall_clock_tflops_per_device` | `xprof_tflops_per_chip`<br>`xprof_tflops_per_device` | Achieved compute throughput per physical TPU chip or per logical device ($$10^{12}$$ FLOPS). |
| **Bandwidth (`GB/s`)** | `wall_clock_bandwidth_per_chip_gb_s`<br>`wall_clock_bandwidth_per_device_gb_s`<br>`wall_clock_bandwidth_per_host_gb_s` | `xprof_bandwidth_per_chip_gb_s`<br>`xprof_bandwidth_per_device_gb_s`<br>`xprof_bandwidth_per_host_gb_s` | Achieved HBM, PCIe, ICI link, or collective bus bandwidth per chip, per device, or per host ($$10^9$$ bytes/sec). |
| **Roofline Efficiency (`%`)** | `wall_clock_compute_roofline_efficiency_pct`<br>`wall_clock_memory_roofline_efficiency_pct`<br>`wall_clock_ici_roofline_efficiency_pct`<br>`wall_clock_roofline_efficiency_pct` | `xprof_compute_roofline_efficiency_pct`<br>`xprof_memory_roofline_efficiency_pct`<br>`xprof_ici_roofline_efficiency_pct`<br>`xprof_roofline_efficiency_pct` | Achieved compute, memory, ICI (`all_to_all` only), or device-to-device link (`device_to_device` only) performance as a percentage of the hardware's theoretical peak. |
| **Roofline Efficiency (`%`)** | `wall_clock_compute_roofline_efficiency_pct`<br>`wall_clock_memory_roofline_efficiency_pct`<br>`wall_clock_ici_roofline_efficiency_pct`<br>`wall_clock_roofline_efficiency_pct` | `xprof_compute_roofline_efficiency_pct`<br>`xprof_memory_roofline_efficiency_pct`<br>`xprof_ici_roofline_efficiency_pct`<br>`xprof_roofline_efficiency_pct` | Achieved compute, memory, ICI (`all_to_all` only), device-to-device link (`device_to_device`: ICI or die-to-die), or host link (`host_to_device` / `device_to_host`: PCIe) performance as a percentage of the hardware's theoretical peak. |

See **[Throughput & Bandwidth Metric Conventions in `docs/BENCHMARKS.md`](docs/BENCHMARKS.md#per-device-vs-per-chip-scaling)** for how `_per_chip` vs. `_per_device` scales across dual-device (`tpu7x`) and single-device (`v6e`) TPU chips and which metrics each benchmark reports.

Expand Down
22 changes: 21 additions & 1 deletion docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ The table below summarizes which `<metric_type>` and `<hardware_scope>` each ben
| :--- | :--- | :--- | :--- |
| **`gemm`** | **Compute Throughput (`TFLOPS`)** & Roofline (`%`) | **Both Per-Chip & Per-Device**<br>(`*_tflops_per_chip`, `*_tflops_per_device`) | **Measures** per-device and aggregate per-chip matrix compute throughput (`TFLOPS`), and **compares** against peak chip compute rooflines (`%`). |
| **`hbm`** | **Memory Bandwidth (`GB/s`)** & Roofline (`%`) | **All Three Scopes (`Per-Device`, `Per-Chip`, `Per-Host`; active measured, smaller derived, larger `None`)**<br>(`*_bandwidth_per_{device,chip,host}_gb_s`) | **Measures** HBM memory bandwidth (`GB/s`) across the selected hardware `scope` (`device` default, `chip`, or `host`), **derives** smaller scopes as the average share of the concurrent aggregate measurement, leaves larger scopes as `None` (no upward extrapolation), and **compares** against peak per-device HBM bandwidth rooflines (`%`). |
| **`host_to_device` / `device_to_host`** | **Host I/O Bandwidth (`GB/s`)** | **All Three Scopes (`Per-Device`, `Per-Chip`, `Per-Host`; active measured, smaller derived, larger `None`)**<br>(`*_bandwidth_per_{device,chip,host}_gb_s`) | **Measures** PCIe transfer bandwidth (`GB/s`) between the host CPU and the selected hardware `scope`: all local devices concurrently (`scope: "host"`, default), one physical chip (`scope: "chip"`), or one local device (`scope: "device"`), **deriving** smaller scopes as the average share while leaving larger scopes as `None` (since shared PCIe links do not scale linearly across cores). |
| **`host_to_device` / `device_to_host`** | **Host I/O Bandwidth (`GB/s`)** & Roofline (`%`) | **All Three Scopes (`Per-Device`, `Per-Chip`, `Per-Host`; active measured, smaller derived, larger `None`)**<br>(`*_bandwidth_per_{device,chip,host}_gb_s`) | **Measures** PCIe transfer bandwidth (`GB/s`) between the host CPU and the selected hardware `scope`: all local devices concurrently (`scope: "host"`, default), one physical chip (`scope: "chip"`), or one local device (`scope: "device"`), **deriving** smaller scopes as the average share while leaving larger scopes as `None` (since shared PCIe links do not scale linearly across cores), and **compares** the measured scope against the host link (PCIe) peak of the chips involved (`%`). |
| **`device_to_device`** | **Device-to-Device Link Bandwidth (`GB/s`)** & Roofline (`%`) | **Per-Device**<br>(`*_bandwidth_per_device_gb_s`) | **Measures** transfer bandwidth (`GB/s`) between each device pair and **compares** it against the peak of the link used, ICI or die-to-die (`%`). |
| **`all_reduce` / `all_gather` / `all_to_all`** | **Collective Bus Bandwidth (`GB/s`)** | **Per-Chip**<br>(`*_bandwidth_per_chip_gb_s`) | **Measures** aggregate external ICI collective bus bandwidth (`GB/s`) per physical TPU chip across the active device mesh. |

Expand Down Expand Up @@ -415,6 +415,26 @@ inputs `A[M, K]` and `B[K, N]` dynamically quantized row-wise/column-wise to
(`-` in console tables, empty in `summary.csv`, `null` in `detailed.json`)
because shared PCIe links (such as a dual-die TPU chip sharing one PCIe
endpoint) cannot be extrapolated upward from a smaller scope.
- **Host Link Roofline** (summary.csv / detailed.json only; not in the
console table):
- `roofline_bound_by` — Link that carries the transfer: always `pcie` (the
chip's host link). Shared with `device_to_device` (`ici` / `die_to_die`),
so every `peak_link_bandwidth_per_*_gb_s` row names its link.
- `peak_link_bandwidth_per_{device|chip|host}_gb_s` — Theoretical host link
peak for the measured `scope` only (`GB/s`): the chip's raw per-direction
PCIe bandwidth (`HardwareSpec.pcie`, e.g. Gen5 x8 = 32 GB/s on `v6e`,
Gen5 x16 = 64 GB/s on `tpu7x`) times the number of chips the measured
devices span (`ceil(num_devices / devices_per_chip)`). The link belongs to
the chip and is shared by its devices, so a single device transferring
alone (`scope="device"`) is bounded by the full chip link. Omitted when the
hardware spec has no PCIe bandwidth.
- `{wall_clock|xprof}_roofline_efficiency_pct` — Measured bandwidth of the
active `scope` as a percentage of that peak (`%`). Emitted for both
`transfer_backend` values: `tpu_sync` typically reaches ~88–92% (the
practical PCIe ceiling after 128b/130b encoding and TLP/DLLP protocol
overhead), while lower `jax` values reflect host staging overhead rather
than the link. Host-side limits (root complex, NUMA, host DRAM) are not
modeled.
- **Latency (`ms`)**:
- `{wall_clock|xprof}_{p50|p90|avg|std}_ms` — Iteration execution latency across median (`p50`), 90th percentile (`p90`), mean (`avg`), and standard deviation (`std`) in milliseconds (`ms`).
- **Benchmark Attributes**:
Expand Down
32 changes: 32 additions & 0 deletions src/accelerator_microbenchmarks/benchmarks/host_device.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
"""Host-to-Device and Device-to-Host transfer performance benchmarks."""

import dataclasses
import logging
import math
from typing import Any, Callable, Optional, Sequence

from accelerator_microbenchmarks.core import base
Expand Down Expand Up @@ -81,6 +83,9 @@ class _HostDeviceBenchmark(base.BaseBenchmark[HostDeviceParams]):
derive_chip_bandwidth: bool = False
# Data moves between host buffers and each local device only; no ICI.
requires_ici: bool = False
roofline_mode: constants.RooflineMode = (
constants.RooflineMode.HOST_DEVICE_LINK
)
REPORT_SCHEMA: Sequence[tuple[str, Callable[[Any], str]]] = (
("dtype", report.format_str),
("transfer_backend", report.format_str),
Expand Down Expand Up @@ -225,6 +230,33 @@ def get_total_bytes(self) -> float:
def get_workload_metadata(self) -> dict[str, Any]:
return {"num_devices": self.num_transfer_devices}

def get_peak_host_link_bandwidth(self) -> float | None:
"""Returns the host link peak (GB/s) for the devices at `config.scope`.

The host PCIe link belongs to the chip and is shared by all of its
devices, so the peak is the per-chip link bandwidth times the number of
chips the measured devices span (`ceil(num_devices / devices_per_chip)`,
the same `devices_per_chip` the scope bandwidths are derived with). A
single device transferring alone (`scope="device"`) is therefore bounded
by its full chip link. The peak applies to both transfer backends;
host-side limits (root complex, NUMA, host DRAM) are not modeled.

Returns:
The peak bandwidth in decimal GB/s, or None (with a warning) when the
hardware spec has no PCIe host link.
"""
pcie = self.hardware_spec.pcie if self.hardware_spec is not None else None
if pcie is None:
logging.warning(
"No PCIe host link bandwidth in the hardware spec; skipping"
" host-link roofline analysis."
)
return None
num_chips = math.ceil(
len(self.get_devices_to_measure()) / self.hardware_spec.devices_per_chip
)
return pcie.bandwidth_per_direction_gb_s * num_chips

def calculate_throughput_metrics(
self, latency_ms: float, prefix: constants.TimingDomain
) -> dict[str, Any]:
Expand Down
14 changes: 14 additions & 0 deletions src/accelerator_microbenchmarks/core/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -571,6 +571,20 @@ def get_peak_link_bandwidth(
" RooflineMode.NETWORK_LINK."
)

def get_peak_host_link_bandwidth(self) -> float | None:
"""Return the host link peak (GB/s) at the active measurement scope.

Required when roofline_mode == RooflineMode.HOST_DEVICE_LINK. The value is
compared against `*_bandwidth_per_<scope>_gb_s` for `config.scope`.
Returns None when the peak cannot be modeled for the current
configuration.
"""
raise NotImplementedError(
f"{self.__class__.__name__} must implement"
" get_peak_host_link_bandwidth() when roofline_mode =="
" RooflineMode.HOST_DEVICE_LINK."
)

@abc.abstractmethod
def get_total_bytes(self) -> float:
"""Calculate total bytes moved to/from HBM."""
Expand Down
9 changes: 8 additions & 1 deletion src/accelerator_microbenchmarks/core/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,23 +48,30 @@ class RooflineMode(str, enum.Enum):
the dies of one chip), as supplied by the benchmark. Emits
`peak_link_bandwidth_per_device_gb_s`, `roofline_bound_by` (a
`TransferLink` value), and `<domain>_roofline_efficiency_pct`.
HOST_DEVICE_LINK: Host<->device transfer roofline: the measured bandwidth
at the active measurement scope is compared against the host link peak
of the chips involved, as supplied by the benchmark. Emits
`peak_link_bandwidth_per_<scope>_gb_s`, `roofline_bound_by`
(`TransferLink.PCIE`), and `<domain>_roofline_efficiency_pct`.
"""

NONE = "none"
COMPUTE = "compute"
MEMORY_HBM = "memory_hbm"
NETWORK_ICI = "network_ici"
NETWORK_LINK = "network_link"
HOST_DEVICE_LINK = "host_device_link"


class TransferLink(enum.StrEnum):
"""Physical link bounding a device-to-device transfer.
"""Physical link bounding a device-to-device or host<->device transfer.

The values are the strings emitted in the `roofline_bound_by` metric.
"""

ICI = "ici"
DIE_TO_DIE = "die_to_die"
PCIE = "pcie"


class TimingDomain(enum.StrEnum):
Expand Down
19 changes: 19 additions & 0 deletions src/accelerator_microbenchmarks/core/roofline.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ def apply_roofline_analysis(
collectives report exclusively `*_bandwidth_per_chip_gb_s`.
`<domain>_roofline_efficiency_pct` (NETWORK_LINK) is defined per device,
since device-to-device transfers report `*_bandwidth_per_device_gb_s`.
`<domain>_roofline_efficiency_pct` (HOST_DEVICE_LINK) is computed at the
measured `config.scope` only, never from derived smaller-scope shares.

Args:
benchmark_instance: The benchmark instance being analyzed.
Expand Down Expand Up @@ -136,4 +138,21 @@ def apply_roofline_analysis(
) * 100.0
return metrics

if mode == constants.RooflineMode.HOST_DEVICE_LINK:
peak_bw = benchmark_instance.get_peak_host_link_bandwidth()
if peak_bw is None or peak_bw <= 0:
return metrics
scope = constants.Scope(benchmark_instance.config.scope).value
metrics[f"peak_link_bandwidth_per_{scope}_gb_s"] = peak_bw
# Shares `peak_link_bandwidth_per_device_gb_s` with NETWORK_LINK, so name
# the bounding link to keep the column self-describing.
metrics["roofline_bound_by"] = str(constants.TransferLink.PCIE)
for prefix in constants.TimingDomain:
actual_bw = metrics.get(f"{prefix}_bandwidth_per_{scope}_gb_s")
if actual_bw is not None and actual_bw >= 0:
metrics[f"{prefix}_roofline_efficiency_pct"] = (
actual_bw / peak_bw
) * 100.0
return metrics

return metrics
1 change: 1 addition & 0 deletions src/accelerator_microbenchmarks/core/system.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
TflopsSpec = schema.TflopsSpec
IciSpec = schema.IciSpec
DieToDieSpec = schema.DieToDieSpec
PcieSpec = schema.PcieSpec
WraparoundRule = schema.WraparoundRule
HbmSpec = schema.HbmSpec

Expand Down
18 changes: 18 additions & 0 deletions src/accelerator_microbenchmarks/specs/schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,21 @@ class DieToDieSpec:
bandwidth_per_direction_gb_s: float


@dataclasses.dataclass(frozen=True)
class PcieSpec:
"""PCIe host link of one chip (theoretical peak).

Attributes:
bandwidth_per_direction_gb_s: Raw bandwidth of the chip's host PCIe link in
one direction, in decimal GB/s (transfer rate x lanes / 8, before
128b/130b line encoding). The link is full-duplex, so each direction
carries this bandwidth independently. All devices of a multi-device chip
share the link.
"""

bandwidth_per_direction_gb_s: float


@dataclasses.dataclass(frozen=True)
class HbmSpec:
"""High Bandwidth Memory specifications (per-chip).
Expand All @@ -144,6 +159,9 @@ class HardwareSpec:
# `None` when the chip has no die-to-die link (one device per chip) or its
# bandwidth has not been verified yet.
die_to_die: DieToDieSpec | None = None
# `None` when the host link is not PCIe or its bandwidth has not been
# verified yet.
pcie: PcieSpec | None = None
hbm: HbmSpec | None = None

@property
Expand Down
3 changes: 3 additions & 0 deletions src/accelerator_microbenchmarks/specs/tpu7x.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@
# (https://docs.cloud.google.com/tpu/docs/tpu7x#dual-chiplet_architecture).
# The D2D link is full duplex.
die_to_die=schema.DieToDieSpec(bandwidth_per_direction_gb_s=800.0),
# Host interface: PCIe Gen5 x16 per chip, shared by both chiplets; raw
# 32 GT/s x 16 lanes / 8.
pcie=schema.PcieSpec(bandwidth_per_direction_gb_s=64.0),
hbm=schema.HbmSpec(peak_bw_gb_s=7380.0),
)
3 changes: 3 additions & 0 deletions src/accelerator_microbenchmarks/specs/v6e.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,8 @@
link_bw_per_direction_gb_s=100.0,
wraparound_rule=schema.WraparoundRule.PER_DIM_GE_16,
),
# Host interface: PCIe Gen5 x8 per chip
# (https://docs.cloud.google.com/tpu/docs/v6e), raw 32 GT/s x 8 lanes / 8.
pcie=schema.PcieSpec(bandwidth_per_direction_gb_s=32.0),
hbm=schema.HbmSpec(peak_bw_gb_s=1638.4),
)
Loading
Loading