Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
| **Compute** | **Matrix Multiplication (GEMM)** | Dense matrix multiplication ($$C = \alpha (A \times B) + \beta C$$) across configurable data types | • Compute Throughput (**TFLOPS**)<br>• Compute Roofline Efficiency (**%**)<br>• Compute Latency (**ms**) |
| **Memory (HBM)** | **HBM Memory Bandwidth** | High Bandwidth Memory STREAM operations (Copy, Scale, Add, Triad, Read-Only, Write-Only) | • Memory Bandwidth (**GB/s**)<br>• Memory Roofline Efficiency (**%**)<br>• Memory Access Latency (**ms**) |
| **Host I/O (PCIe)** | **Host-to-Device (H2D)**<br>**Device-to-Host (D2H)** | Host CPU memory to/from accelerator HBM data transfers | • PCIe Transfer Bandwidth (**GB/s**)<br>• Transfer Latency (**ms**) |
| **Interconnect (ICI)** | **Device-to-Device (D2D)** | Point-to-point inter-chip data transfers across ICI links | • ICI Link Bandwidth (**GB/s**)<br>• Pairwise N × N Device Bandwidth Matrix<br>• Transfer Latency (**ms**) |
| **Interconnect (ICI)** | **Device-to-Device (D2D)** | Data transfers between each device pair, over ICI (different chips) or die-to-die (same chip) | • D2D Link Bandwidth (**GB/s**)<br>• Pairwise N × N Device Bandwidth Matrix<br>• Transfer Latency (**ms**) |
| **Collectives** | **All-Gather**<br>**All-Reduce**<br>**All-to-All** | Distributed collective communication across multi-chip topologies | • Collective Bus Bandwidth (**GB/s**)<br>• Collective Step Latency (**ms**) |

---
Expand Down Expand Up @@ -210,7 +210,7 @@ tpums benchmark run host_to_device --xprof_timing --data_size_mib 256 --dtype bf
# 4. Device-to-Host (PCIe) Transfer Latency & Bandwidth on all local devices
tpums benchmark run device_to_host --xprof_timing --data_size_mib 256 --dtype bfloat16 --transfer_backend tpu_sync --scope host

# 5. Device-to-Device (ICI) point-to-point transfer across all pairs
# 5. Device-to-device (D2D) transfer across all pairs
tpums benchmark run device_to_device --xprof_timing --data_size_mib 1024 --direction uni --dtype bfloat16

# 6. Multi-Device All-Reduce Collective
Expand Down Expand Up @@ -487,7 +487,7 @@ TPUMS records primary performance metrics across both timing domains:
| **Latency (`ms`)** | `wall_clock_p50_ms`<br>*(+ `p90`, `avg`, `std`)* | `xprof_p50_ms`<br>*(+ `p90`, `avg`, `std`)* | Execution duration across timed iterations (`p50` median, `p90`, mean, and standard deviation). |
| **Compute Throughput (`TFLOPS`)** | `wall_clock_tflops_per_chip`<br>`wall_clock_tflops_per_device` | `xprof_tflops_per_chip`<br>`xprof_tflops_per_device` | Achieved compute throughput per physical TPU chip or per logical device ($$10^{12}$$ FLOPS). |
| **Bandwidth (`GB/s`)** | `wall_clock_bandwidth_per_chip_gb_s`<br>`wall_clock_bandwidth_per_device_gb_s`<br>`wall_clock_bandwidth_per_host_gb_s` | `xprof_bandwidth_per_chip_gb_s`<br>`xprof_bandwidth_per_device_gb_s`<br>`xprof_bandwidth_per_host_gb_s` | Achieved HBM, PCIe, ICI link, or collective bus bandwidth per chip, per device, or per host ($$10^9$$ bytes/sec). |
| **Roofline Efficiency (`%`)** | `wall_clock_compute_roofline_efficiency_pct`<br>`wall_clock_memory_roofline_efficiency_pct`<br>`wall_clock_ici_roofline_efficiency_pct` | `xprof_compute_roofline_efficiency_pct`<br>`xprof_memory_roofline_efficiency_pct`<br>`xprof_ici_roofline_efficiency_pct` | Achieved compute, memory, or ICI (`all_to_all` only) performance as a percentage of the hardware's theoretical peak (`0–100%`). |
| **Roofline Efficiency (`%`)** | `wall_clock_compute_roofline_efficiency_pct`<br>`wall_clock_memory_roofline_efficiency_pct`<br>`wall_clock_ici_roofline_efficiency_pct`<br>`wall_clock_roofline_efficiency_pct` | `xprof_compute_roofline_efficiency_pct`<br>`xprof_memory_roofline_efficiency_pct`<br>`xprof_ici_roofline_efficiency_pct`<br>`xprof_roofline_efficiency_pct` | Achieved compute, memory, ICI (`all_to_all` only), or device-to-device link (`device_to_device` only) performance as a percentage of the hardware's theoretical peak. |

See **[Throughput & Bandwidth Metric Conventions in `docs/BENCHMARKS.md`](docs/BENCHMARKS.md#per-device-vs-per-chip-scaling)** for how `_per_chip` vs. `_per_device` scales across dual-device (`tpu7x`) and single-device (`v6e`) TPU chips and which metrics each benchmark reports.

Expand Down
2 changes: 1 addition & 1 deletion automation/report_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@
),
"device_to_device": (
"🔗 Device-to-Device (D2D)",
"Peer-to-peer inter-chip transfer bandwidth and latency",
"Device-to-device transfer bandwidth and latency",
),
}

Expand Down
19 changes: 13 additions & 6 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ This guide provides the reference for **TPUMS (TPU Microbenchmark Suite)** param
- [2.1.3 GEMM Pulse (`gemm_pulse`) [experimental]](#213-gemm-pulse-gemm_pulse-experimental)
- [2.2 HBM Memory Bandwidth (`hbm`)](#22-hbm-memory-bandwidth-hbm)
- [2.3 Host I/O Bandwidth (`host_to_device`, `device_to_host`)](#23-host-io-bandwidth-host_to_device-device_to_host)
- [2.4 Inter-Chip Interconnect (`device_to_device`)](#24-inter-chip-interconnect-device_to_device)
- [2.4 Device-to-Device Transfers (`device_to_device`)](#24-inter-chip-interconnect-device_to_device)
- [2.5 Collective Communications (`all_reduce`, `all_gather`, `all_to_all`)](#25-collective-communications-all_reduce-all_gather-all_to_all)
- [Collective Concepts: `mesh_shape`, `sharding_strategy` & Parallel vs. Non-Parallel Replica Groups](#collective-concepts)

Expand Down Expand Up @@ -101,7 +101,7 @@ The table below summarizes which `<metric_type>` and `<hardware_scope>` each ben
| **`gemm`** | **Compute Throughput (`TFLOPS`)** & Roofline (`%`) | **Both Per-Chip & Per-Device**<br>(`*_tflops_per_chip`, `*_tflops_per_device`) | **Measures** per-device and aggregate per-chip matrix compute throughput (`TFLOPS`), and **compares** against peak chip compute rooflines (`%`). |
| **`hbm`** | **Memory Bandwidth (`GB/s`)** & Roofline (`%`) | **All Three Scopes (`Per-Device`, `Per-Chip`, `Per-Host`; active measured, smaller derived, larger `None`)**<br>(`*_bandwidth_per_{device,chip,host}_gb_s`) | **Measures** HBM memory bandwidth (`GB/s`) across the selected hardware `scope` (`device` default, `chip`, or `host`), **derives** smaller scopes as the average share of the concurrent aggregate measurement, leaves larger scopes as `None` (no upward extrapolation), and **compares** against peak per-device HBM bandwidth rooflines (`%`). |
| **`host_to_device` / `device_to_host`** | **Host I/O Bandwidth (`GB/s`)** | **All Three Scopes (`Per-Device`, `Per-Chip`, `Per-Host`; active measured, smaller derived, larger `None`)**<br>(`*_bandwidth_per_{device,chip,host}_gb_s`) | **Measures** PCIe transfer bandwidth (`GB/s`) between the host CPU and the selected hardware `scope`: all local devices concurrently (`scope: "host"`, default), one physical chip (`scope: "chip"`), or one local device (`scope: "device"`), **deriving** smaller scopes as the average share while leaving larger scopes as `None` (since shared PCIe links do not scale linearly across cores). |
| **`device_to_device`** | **ICI Link Bandwidth (`GB/s`)** | **Per-Device**<br>(`*_bandwidth_per_device_gb_s`) | **Measures** point-to-point ICI link bandwidth (`GB/s`) between each logical device pair `(src_device_index, dst_device_index)`. |
| **`device_to_device`** | **Device-to-Device Link Bandwidth (`GB/s`)** & Roofline (`%`) | **Per-Device**<br>(`*_bandwidth_per_device_gb_s`) | **Measures** transfer bandwidth (`GB/s`) between each device pair and **compares** it against the peak of the link used, ICI or die-to-die (`%`). |
| **`all_reduce` / `all_gather` / `all_to_all`** | **Collective Bus Bandwidth (`GB/s`)** | **Per-Chip**<br>(`*_bandwidth_per_chip_gb_s`) | **Measures** aggregate external ICI collective bus bandwidth (`GB/s`) per physical TPU chip across the active device mesh. |

---
Expand Down Expand Up @@ -408,9 +408,9 @@ metrics when `enable_checksum=True`.
---

<a id="24-inter-chip-interconnect-device_to_device"></a>
### 2.4 Inter-Chip Interconnect (`device_to_device`)
### 2.4 Device-to-Device Transfers (`device_to_device`)

- **Operation**: Measures point-to-point Inter-Chip Interconnect (ICI) link bandwidth (**GB/s**) and latency (**ms**) across all **`N × (N - 1)` directional device pairs** `(src_device_index, dst_device_index)` in the active topology (`src != dst`), supporting both single-host and multi-host topologies.
- **Operation**: Measures device-to-device transfer bandwidth (**GB/s**) and latency (**ms**) across all **`N × (N - 1)` directional device pairs** `(src_device_index, dst_device_index)` in the active topology (`src != dst`), supporting both single-host and multi-host topologies. Pairs on different chips transfer over the Inter-Chip Interconnect (ICI); on chips with two devices (such as **`tpu7x`**), the two devices of one chip transfer over the on-package die-to-die link.

#### Parameters

Expand All @@ -423,15 +423,22 @@ metrics when `enable_checksum=True`.

#### Reported Metrics

- **ICI Link Bandwidth**:
- `{wall_clock|xprof}_bandwidth_per_device_gb_s` — Point-to-point link bandwidth between `(src_device_index, dst_device_index)` (`GB/s`), also formatted in `stdout` as an **`N × N` pairwise device bandwidth matrix** (`src` rows × `dst` columns).
- **Link Bandwidth**:
- `{wall_clock|xprof}_bandwidth_per_device_gb_s` — Device-to-device transfer bandwidth between `(src_device_index, dst_device_index)` (`GB/s`), also formatted in `stdout` as an **`N × N` pairwise device bandwidth matrix** (`src` rows × `dst` columns). For `"bi"`, the bytes of both directions are counted.
- **Link Roofline**:
- `roofline_bound_by` — Link that carries the transfer: `ici` (devices on different chips) or `die_to_die` (the two devices of one chip, e.g. `tpu7x`).
- `peak_link_bandwidth_per_device_gb_s` — Theoretical peak bandwidth of that link (`GB/s`), doubled for `"bi"`.
- `{wall_clock|xprof}_roofline_efficiency_pct` — Measured bandwidth as a percentage of the peak (`%`).
- **Latency (`ms`)**:
- `{wall_clock|xprof}_{p50|p90|avg|std}_ms` — Iteration execution latency across median (`p50`), 90th percentile (`p90`), mean (`avg`), and standard deviation (`std`) in milliseconds (`ms`).
- **Benchmark Attributes**:
- `src_device_index` / `dst_device_index` — Source and destination logical device indices for the measured pair.
- `direction` — Active transfer direction (`"uni"` or `"bi"`).
- `data_size_mib` — Payload size per directional stream (`MiB`).

> [!NOTE]
> Only one device pair transfers at a time, so each transfer gets the full link. ICI links are shared by the devices of a chip, so don't sum ICI peaks per chip.

---

<a id="25-collective-communications-all_reduce-all_gather-all_to_all"></a>
Expand Down
110 changes: 110 additions & 0 deletions src/accelerator_microbenchmarks/benchmarks/device_to_device.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
"""Device-to-Device (D2D) transfer performance microbenchmark."""

import dataclasses
import logging
from typing import Any, Callable, Optional, Sequence

from accelerator_microbenchmarks.core import base
from accelerator_microbenchmarks.core import constants
from accelerator_microbenchmarks.core import dtypes
from accelerator_microbenchmarks.core import platform
from accelerator_microbenchmarks.core import registry
from accelerator_microbenchmarks.core import report
from accelerator_microbenchmarks.core import system
Expand All @@ -22,6 +24,85 @@ class TransferDirection(constants.ParamEnum):
BI = "bi"


def classify_transfer_link(
src: jax.Device, dst: jax.Device
) -> constants.TransferLink:
"""Returns the physical link that carries a `src -> dst` transfer.

Devices on the same chip share physical chip coordinates and talk over the
on-package die-to-die link; devices on different chips of the same slice
talk over ICI.

Args:
src: Source device.
dst: Destination device.

Returns:
`TransferLink.DIE_TO_DIE` if both devices are on the same chip, otherwise
`TransferLink.ICI`.

Raises:
ValueError: If the devices have no physical chip coordinates (non-TPU) or
are on different slices (the transfer then crosses DCN).
"""
src_coords = getattr(src, "coords", None)
dst_coords = getattr(dst, "coords", None)
if src_coords is None or dst_coords is None:
raise ValueError("devices have no physical chip coordinates (non-TPU)")
if platform.get_device_slice_index(src) != platform.get_device_slice_index(
dst
):
raise ValueError("src and dst are on different slices (DCN)")
if tuple(src_coords) == tuple(dst_coords):
return constants.TransferLink.DIE_TO_DIE
return constants.TransferLink.ICI


def device_to_device_peak_bandwidth_gb_s(
link: constants.TransferLink,
direction: TransferDirection,
hardware_spec: system.HardwareSpec,
) -> float:
"""Returns the theoretical peak bandwidth of one D2D transfer (GB/s).

A device-to-device transfer is bounded by a single physical link per
direction. For ICI this holds independent of the hop count: a single
transfer follows one route, so measured cross-chip transfers saturate one
link whether the chips are neighbors or several hops apart. Links are
full-duplex and `DeviceToDeviceBenchmark.get_total_bytes()` counts the
bytes of both directions for `BI`, so the bidirectional peak is twice the
per-direction peak.

Scope: the result is the ceiling for the single measured `src -> dst`
transfer, which runs alone (one device pair at a time) and so has the whole
link to itself. It is not a per-device share of a chip resource: ICI links
belong to the chip and are shared by all of its devices, so ICI peaks must
not be summed across the devices of a chip.

Args:
link: The link carrying the transfer.
direction: Unidirectional or bidirectional transfer.
hardware_spec: Hardware spec holding the link bandwidths.

Returns:
The peak bandwidth in decimal GB/s, in the same units as
`*_bandwidth_per_device_gb_s`.

Raises:
ValueError: If `hardware_spec` has no bandwidth for `link`.
"""
if link == constants.TransferLink.DIE_TO_DIE:
if hardware_spec.die_to_die is None:
raise ValueError(f"no die-to-die spec for {hardware_spec.name}")
per_direction_gb_s = hardware_spec.die_to_die.bandwidth_per_direction_gb_s
else:
if hardware_spec.ici is None:
raise ValueError(f"no ICI spec for {hardware_spec.name}")
per_direction_gb_s = hardware_spec.ici.link_bw_per_direction_gb_s
num_directions = 2.0 if direction == TransferDirection.BI else 1.0
return per_direction_gb_s * num_directions


@dataclasses.dataclass
class DeviceToDeviceParams(base.SingleDtypeBenchmarkParams):
"""YAML / CLI configuration specification (NO src/dst fields)."""
Expand Down Expand Up @@ -93,6 +174,7 @@ class DeviceToDeviceBenchmark(base.BaseBenchmark[DeviceToDeviceTestCaseParams]):

Config = DeviceToDeviceParams
derive_chip_bandwidth: bool = False
roofline_mode: constants.RooflineMode = constants.RooflineMode.NETWORK_LINK
REPORT_SCHEMA: Sequence[tuple[str, Callable[[Any], str]]] = (
("dtype", report.format_str),
("direction", report.format_str),
Expand Down Expand Up @@ -234,3 +316,31 @@ def calculate_throughput_metrics(
return {
f"{prefix}_bandwidth_per_device_gb_s": bandwidth_gb_s,
}

def get_peak_link_bandwidth(
self,
) -> tuple[float, constants.TransferLink] | None:
"""Returns the D2D peak and its bounding link, or None (with a warning).

The peak is not modeled (and the roofline columns are omitted) for:
- Non-TPU devices without chip coordinates.
- Pairs on different slices (DCN).
- Hardware specs without a bandwidth for the bounding link.
"""
if self.mesh is None:
raise ValueError("Mesh not initialized.")
# Indices refer to mesh positions, which need not follow chip order, so
# classify the actual devices rather than the indices.
src = self.mesh.devices.flat[self.config.src_device_index]
dst = self.mesh.devices.flat[self.config.dst_device_index]
try:
link = classify_transfer_link(src, dst)
peak_gb_s = device_to_device_peak_bandwidth_gb_s(
link, self.config.direction, self.hardware_spec
)
except ValueError as e:
logging.warning(
"Skipping device_to_device *_roofline_efficiency_pct: %s.", e
)
return None
return peak_gb_s, link
14 changes: 14 additions & 0 deletions src/accelerator_microbenchmarks/core/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -557,6 +557,20 @@ def get_peak_ici_bandwidth_per_chip(self) -> float | None:
" RooflineMode.NETWORK_ICI."
)

def get_peak_link_bandwidth(
self,
) -> tuple[float, constants.TransferLink] | None:
"""Return the peak link bandwidth per device (GB/s) and the bounding link.

Required when roofline_mode == RooflineMode.NETWORK_LINK. Returns None
when the peak cannot be modeled for the current configuration.
"""
raise NotImplementedError(
f"{self.__class__.__name__} must implement"
" get_peak_link_bandwidth() when roofline_mode =="
" RooflineMode.NETWORK_LINK."
)

@abc.abstractmethod
def get_total_bytes(self) -> float:
"""Calculate total bytes moved to/from HBM."""
Expand Down
17 changes: 17 additions & 0 deletions src/accelerator_microbenchmarks/core/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,29 @@ class RooflineMode(str, enum.Enum):
supplied by the benchmark (currently only `all_to_all`, whose peak is
bounded by bisection bandwidth). Emits `peak_ici_bandwidth_per_chip_gb_s`
and `<domain>_ici_roofline_efficiency_pct`.
NETWORK_LINK: Device-to-device transfer roofline: the measured per-device
bandwidth is compared against the peak of the single physical link that
carries the transfer (ICI between chips, or the die-to-die link between
the dies of one chip), as supplied by the benchmark. Emits
`peak_link_bandwidth_per_device_gb_s`, `roofline_bound_by` (a
`TransferLink` value), and `<domain>_roofline_efficiency_pct`.
"""

NONE = "none"
COMPUTE = "compute"
MEMORY_HBM = "memory_hbm"
NETWORK_ICI = "network_ici"
NETWORK_LINK = "network_link"


class TransferLink(enum.StrEnum):
"""Physical link bounding a device-to-device transfer.

The values are the strings emitted in the `roofline_bound_by` metric.
"""

ICI = "ici"
DIE_TO_DIE = "die_to_die"


class TimingDomain(enum.StrEnum):
Expand Down
17 changes: 17 additions & 0 deletions src/accelerator_microbenchmarks/core/roofline.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ def apply_roofline_analysis(
`<domain>_memory_roofline_efficiency_pct` apply invariantly to both scopes.
`<domain>_ici_roofline_efficiency_pct` is defined per chip only, since ICI
collectives report exclusively `*_bandwidth_per_chip_gb_s`.
`<domain>_roofline_efficiency_pct` (NETWORK_LINK) is defined per device,
since device-to-device transfers report `*_bandwidth_per_device_gb_s`.

Args:
benchmark_instance: The benchmark instance being analyzed.
Expand Down Expand Up @@ -119,4 +121,19 @@ def apply_roofline_analysis(
) * 100.0
return metrics

if mode == constants.RooflineMode.NETWORK_LINK:
peak = benchmark_instance.get_peak_link_bandwidth()
if peak is None:
return metrics
peak_bw, link = peak
metrics["peak_link_bandwidth_per_device_gb_s"] = peak_bw
metrics["roofline_bound_by"] = str(link)
for prefix in constants.TimingDomain:
actual_bw = metrics.get(f"{prefix}_bandwidth_per_device_gb_s")
if actual_bw is not None and actual_bw >= 0 and peak_bw > 0:
metrics[f"{prefix}_roofline_efficiency_pct"] = (
actual_bw / peak_bw
) * 100.0
return metrics

return metrics
1 change: 1 addition & 0 deletions src/accelerator_microbenchmarks/core/system.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
HardwareSpec = schema.HardwareSpec
TflopsSpec = schema.TflopsSpec
IciSpec = schema.IciSpec
DieToDieSpec = schema.DieToDieSpec
WraparoundRule = schema.WraparoundRule
HbmSpec = schema.HbmSpec

Expand Down
Loading
Loading