diff --git a/README.md b/README.md index f6d0a4ba..fc1927ba 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ | **Compute** | **Matrix Multiplication (GEMM)** | Dense matrix multiplication ($$C = \alpha (A \times B) + \beta C$$) across configurable data types | • Compute Throughput (**TFLOPS**)
• Compute Roofline Efficiency (**%**)
• Compute Latency (**ms**) | | **Memory (HBM)** | **HBM Memory Bandwidth** | High Bandwidth Memory STREAM operations (Copy, Scale, Add, Triad, Read-Only, Write-Only) | • Memory Bandwidth (**GB/s**)
• Memory Roofline Efficiency (**%**)
• Memory Access Latency (**ms**) | | **Host I/O (PCIe)** | **Host-to-Device (H2D)**
**Device-to-Host (D2H)** | Host CPU memory to/from accelerator HBM data transfers | • PCIe Transfer Bandwidth (**GB/s**)
• Transfer Latency (**ms**) | -| **Interconnect (ICI)** | **Device-to-Device (D2D)** | Point-to-point inter-chip data transfers across ICI links | • ICI Link Bandwidth (**GB/s**)
• Pairwise N × N Device Bandwidth Matrix
• Transfer Latency (**ms**) | +| **Interconnect (ICI)** | **Device-to-Device (D2D)** | Data transfers between each device pair, over ICI (different chips) or die-to-die (same chip) | • D2D Link Bandwidth (**GB/s**)
• Pairwise N × N Device Bandwidth Matrix
• Transfer Latency (**ms**) | | **Collectives** | **All-Gather**
**All-Reduce**
**All-to-All** | Distributed collective communication across multi-chip topologies | • Collective Bus Bandwidth (**GB/s**)
• Collective Step Latency (**ms**) | --- @@ -210,7 +210,7 @@ tpums benchmark run host_to_device --xprof_timing --data_size_mib 256 --dtype bf # 4. Device-to-Host (PCIe) Transfer Latency & Bandwidth on all local devices tpums benchmark run device_to_host --xprof_timing --data_size_mib 256 --dtype bfloat16 --transfer_backend tpu_sync --scope host -# 5. Device-to-Device (ICI) point-to-point transfer across all pairs +# 5. Device-to-device (D2D) transfer across all pairs tpums benchmark run device_to_device --xprof_timing --data_size_mib 1024 --direction uni --dtype bfloat16 # 6. Multi-Device All-Reduce Collective @@ -487,7 +487,7 @@ TPUMS records primary performance metrics across both timing domains: | **Latency (`ms`)** | `wall_clock_p50_ms`
*(+ `p90`, `avg`, `std`)* | `xprof_p50_ms`
*(+ `p90`, `avg`, `std`)* | Execution duration across timed iterations (`p50` median, `p90`, mean, and standard deviation). | | **Compute Throughput (`TFLOPS`)** | `wall_clock_tflops_per_chip`
`wall_clock_tflops_per_device` | `xprof_tflops_per_chip`
`xprof_tflops_per_device` | Achieved compute throughput per physical TPU chip or per logical device ($$10^{12}$$ FLOPS). | | **Bandwidth (`GB/s`)** | `wall_clock_bandwidth_per_chip_gb_s`
`wall_clock_bandwidth_per_device_gb_s`
`wall_clock_bandwidth_per_host_gb_s` | `xprof_bandwidth_per_chip_gb_s`
`xprof_bandwidth_per_device_gb_s`
`xprof_bandwidth_per_host_gb_s` | Achieved HBM, PCIe, ICI link, or collective bus bandwidth per chip, per device, or per host ($$10^9$$ bytes/sec). | -| **Roofline Efficiency (`%`)** | `wall_clock_compute_roofline_efficiency_pct`
`wall_clock_memory_roofline_efficiency_pct`
`wall_clock_ici_roofline_efficiency_pct` | `xprof_compute_roofline_efficiency_pct`
`xprof_memory_roofline_efficiency_pct`
`xprof_ici_roofline_efficiency_pct` | Achieved compute, memory, or ICI (`all_to_all` only) performance as a percentage of the hardware's theoretical peak (`0–100%`). | +| **Roofline Efficiency (`%`)** | `wall_clock_compute_roofline_efficiency_pct`
`wall_clock_memory_roofline_efficiency_pct`
`wall_clock_ici_roofline_efficiency_pct`
`wall_clock_roofline_efficiency_pct` | `xprof_compute_roofline_efficiency_pct`
`xprof_memory_roofline_efficiency_pct`
`xprof_ici_roofline_efficiency_pct`
`xprof_roofline_efficiency_pct` | Achieved compute, memory, ICI (`all_to_all` only), or device-to-device link (`device_to_device` only) performance as a percentage of the hardware's theoretical peak. | See **[Throughput & Bandwidth Metric Conventions in `docs/BENCHMARKS.md`](docs/BENCHMARKS.md#per-device-vs-per-chip-scaling)** for how `_per_chip` vs. `_per_device` scales across dual-device (`tpu7x`) and single-device (`v6e`) TPU chips and which metrics each benchmark reports. diff --git a/automation/report_generator.py b/automation/report_generator.py index 2e875425..9eb44284 100644 --- a/automation/report_generator.py +++ b/automation/report_generator.py @@ -64,7 +64,7 @@ ), "device_to_device": ( "🔗 Device-to-Device (D2D)", - "Peer-to-peer inter-chip transfer bandwidth and latency", + "Device-to-device transfer bandwidth and latency", ), } diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index fbcd1cd3..ed4f9284 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -20,7 +20,7 @@ This guide provides the reference for **TPUMS (TPU Microbenchmark Suite)** param - [2.1.3 GEMM Pulse (`gemm_pulse`) [experimental]](#213-gemm-pulse-gemm_pulse-experimental) - [2.2 HBM Memory Bandwidth (`hbm`)](#22-hbm-memory-bandwidth-hbm) - [2.3 Host I/O Bandwidth (`host_to_device`, `device_to_host`)](#23-host-io-bandwidth-host_to_device-device_to_host) - - [2.4 Inter-Chip Interconnect (`device_to_device`)](#24-inter-chip-interconnect-device_to_device) + - [2.4 Device-to-Device Transfers (`device_to_device`)](#24-inter-chip-interconnect-device_to_device) - [2.5 Collective Communications (`all_reduce`, `all_gather`, `all_to_all`)](#25-collective-communications-all_reduce-all_gather-all_to_all) - [Collective Concepts: `mesh_shape`, `sharding_strategy` & Parallel vs. Non-Parallel Replica Groups](#collective-concepts) @@ -101,7 +101,7 @@ The table below summarizes which `` and `` each ben | **`gemm`** | **Compute Throughput (`TFLOPS`)** & Roofline (`%`) | **Both Per-Chip & Per-Device**
(`*_tflops_per_chip`, `*_tflops_per_device`) | **Measures** per-device and aggregate per-chip matrix compute throughput (`TFLOPS`), and **compares** against peak chip compute rooflines (`%`). | | **`hbm`** | **Memory Bandwidth (`GB/s`)** & Roofline (`%`) | **All Three Scopes (`Per-Device`, `Per-Chip`, `Per-Host`; active measured, smaller derived, larger `None`)**
(`*_bandwidth_per_{device,chip,host}_gb_s`) | **Measures** HBM memory bandwidth (`GB/s`) across the selected hardware `scope` (`device` default, `chip`, or `host`), **derives** smaller scopes as the average share of the concurrent aggregate measurement, leaves larger scopes as `None` (no upward extrapolation), and **compares** against peak per-device HBM bandwidth rooflines (`%`). | | **`host_to_device` / `device_to_host`** | **Host I/O Bandwidth (`GB/s`)** | **All Three Scopes (`Per-Device`, `Per-Chip`, `Per-Host`; active measured, smaller derived, larger `None`)**
(`*_bandwidth_per_{device,chip,host}_gb_s`) | **Measures** PCIe transfer bandwidth (`GB/s`) between the host CPU and the selected hardware `scope`: all local devices concurrently (`scope: "host"`, default), one physical chip (`scope: "chip"`), or one local device (`scope: "device"`), **deriving** smaller scopes as the average share while leaving larger scopes as `None` (since shared PCIe links do not scale linearly across cores). | -| **`device_to_device`** | **ICI Link Bandwidth (`GB/s`)** | **Per-Device**
(`*_bandwidth_per_device_gb_s`) | **Measures** point-to-point ICI link bandwidth (`GB/s`) between each logical device pair `(src_device_index, dst_device_index)`. | +| **`device_to_device`** | **Device-to-Device Link Bandwidth (`GB/s`)** & Roofline (`%`) | **Per-Device**
(`*_bandwidth_per_device_gb_s`) | **Measures** transfer bandwidth (`GB/s`) between each device pair and **compares** it against the peak of the link used, ICI or die-to-die (`%`). | | **`all_reduce` / `all_gather` / `all_to_all`** | **Collective Bus Bandwidth (`GB/s`)** | **Per-Chip**
(`*_bandwidth_per_chip_gb_s`) | **Measures** aggregate external ICI collective bus bandwidth (`GB/s`) per physical TPU chip across the active device mesh. | --- @@ -408,9 +408,9 @@ metrics when `enable_checksum=True`. --- -### 2.4 Inter-Chip Interconnect (`device_to_device`) +### 2.4 Device-to-Device Transfers (`device_to_device`) -- **Operation**: Measures point-to-point Inter-Chip Interconnect (ICI) link bandwidth (**GB/s**) and latency (**ms**) across all **`N × (N - 1)` directional device pairs** `(src_device_index, dst_device_index)` in the active topology (`src != dst`), supporting both single-host and multi-host topologies. +- **Operation**: Measures device-to-device transfer bandwidth (**GB/s**) and latency (**ms**) across all **`N × (N - 1)` directional device pairs** `(src_device_index, dst_device_index)` in the active topology (`src != dst`), supporting both single-host and multi-host topologies. Pairs on different chips transfer over the Inter-Chip Interconnect (ICI); on chips with two devices (such as **`tpu7x`**), the two devices of one chip transfer over the on-package die-to-die link. #### Parameters @@ -423,8 +423,12 @@ metrics when `enable_checksum=True`. #### Reported Metrics -- **ICI Link Bandwidth**: - - `{wall_clock|xprof}_bandwidth_per_device_gb_s` — Point-to-point link bandwidth between `(src_device_index, dst_device_index)` (`GB/s`), also formatted in `stdout` as an **`N × N` pairwise device bandwidth matrix** (`src` rows × `dst` columns). +- **Link Bandwidth**: + - `{wall_clock|xprof}_bandwidth_per_device_gb_s` — Device-to-device transfer bandwidth between `(src_device_index, dst_device_index)` (`GB/s`), also formatted in `stdout` as an **`N × N` pairwise device bandwidth matrix** (`src` rows × `dst` columns). For `"bi"`, the bytes of both directions are counted. +- **Link Roofline**: + - `roofline_bound_by` — Link that carries the transfer: `ici` (devices on different chips) or `die_to_die` (the two devices of one chip, e.g. `tpu7x`). + - `peak_link_bandwidth_per_device_gb_s` — Theoretical peak bandwidth of that link (`GB/s`), doubled for `"bi"`. + - `{wall_clock|xprof}_roofline_efficiency_pct` — Measured bandwidth as a percentage of the peak (`%`). - **Latency (`ms`)**: - `{wall_clock|xprof}_{p50|p90|avg|std}_ms` — Iteration execution latency across median (`p50`), 90th percentile (`p90`), mean (`avg`), and standard deviation (`std`) in milliseconds (`ms`). - **Benchmark Attributes**: @@ -432,6 +436,9 @@ metrics when `enable_checksum=True`. - `direction` — Active transfer direction (`"uni"` or `"bi"`). - `data_size_mib` — Payload size per directional stream (`MiB`). +> [!NOTE] +> Only one device pair transfers at a time, so each transfer gets the full link. ICI links are shared by the devices of a chip, so don't sum ICI peaks per chip. + --- diff --git a/src/accelerator_microbenchmarks/benchmarks/device_to_device.py b/src/accelerator_microbenchmarks/benchmarks/device_to_device.py index bc1d8e97..7dbd3366 100644 --- a/src/accelerator_microbenchmarks/benchmarks/device_to_device.py +++ b/src/accelerator_microbenchmarks/benchmarks/device_to_device.py @@ -1,11 +1,13 @@ """Device-to-Device (D2D) transfer performance microbenchmark.""" import dataclasses +import logging from typing import Any, Callable, Optional, Sequence from accelerator_microbenchmarks.core import base from accelerator_microbenchmarks.core import constants from accelerator_microbenchmarks.core import dtypes +from accelerator_microbenchmarks.core import platform from accelerator_microbenchmarks.core import registry from accelerator_microbenchmarks.core import report from accelerator_microbenchmarks.core import system @@ -22,6 +24,85 @@ class TransferDirection(constants.ParamEnum): BI = "bi" +def classify_transfer_link( + src: jax.Device, dst: jax.Device +) -> constants.TransferLink: + """Returns the physical link that carries a `src -> dst` transfer. + + Devices on the same chip share physical chip coordinates and talk over the + on-package die-to-die link; devices on different chips of the same slice + talk over ICI. + + Args: + src: Source device. + dst: Destination device. + + Returns: + `TransferLink.DIE_TO_DIE` if both devices are on the same chip, otherwise + `TransferLink.ICI`. + + Raises: + ValueError: If the devices have no physical chip coordinates (non-TPU) or + are on different slices (the transfer then crosses DCN). + """ + src_coords = getattr(src, "coords", None) + dst_coords = getattr(dst, "coords", None) + if src_coords is None or dst_coords is None: + raise ValueError("devices have no physical chip coordinates (non-TPU)") + if platform.get_device_slice_index(src) != platform.get_device_slice_index( + dst + ): + raise ValueError("src and dst are on different slices (DCN)") + if tuple(src_coords) == tuple(dst_coords): + return constants.TransferLink.DIE_TO_DIE + return constants.TransferLink.ICI + + +def device_to_device_peak_bandwidth_gb_s( + link: constants.TransferLink, + direction: TransferDirection, + hardware_spec: system.HardwareSpec, +) -> float: + """Returns the theoretical peak bandwidth of one D2D transfer (GB/s). + + A device-to-device transfer is bounded by a single physical link per + direction. For ICI this holds independent of the hop count: a single + transfer follows one route, so measured cross-chip transfers saturate one + link whether the chips are neighbors or several hops apart. Links are + full-duplex and `DeviceToDeviceBenchmark.get_total_bytes()` counts the + bytes of both directions for `BI`, so the bidirectional peak is twice the + per-direction peak. + + Scope: the result is the ceiling for the single measured `src -> dst` + transfer, which runs alone (one device pair at a time) and so has the whole + link to itself. It is not a per-device share of a chip resource: ICI links + belong to the chip and are shared by all of its devices, so ICI peaks must + not be summed across the devices of a chip. + + Args: + link: The link carrying the transfer. + direction: Unidirectional or bidirectional transfer. + hardware_spec: Hardware spec holding the link bandwidths. + + Returns: + The peak bandwidth in decimal GB/s, in the same units as + `*_bandwidth_per_device_gb_s`. + + Raises: + ValueError: If `hardware_spec` has no bandwidth for `link`. + """ + if link == constants.TransferLink.DIE_TO_DIE: + if hardware_spec.die_to_die is None: + raise ValueError(f"no die-to-die spec for {hardware_spec.name}") + per_direction_gb_s = hardware_spec.die_to_die.bandwidth_per_direction_gb_s + else: + if hardware_spec.ici is None: + raise ValueError(f"no ICI spec for {hardware_spec.name}") + per_direction_gb_s = hardware_spec.ici.link_bw_per_direction_gb_s + num_directions = 2.0 if direction == TransferDirection.BI else 1.0 + return per_direction_gb_s * num_directions + + @dataclasses.dataclass class DeviceToDeviceParams(base.SingleDtypeBenchmarkParams): """YAML / CLI configuration specification (NO src/dst fields).""" @@ -93,6 +174,7 @@ class DeviceToDeviceBenchmark(base.BaseBenchmark[DeviceToDeviceTestCaseParams]): Config = DeviceToDeviceParams derive_chip_bandwidth: bool = False + roofline_mode: constants.RooflineMode = constants.RooflineMode.NETWORK_LINK REPORT_SCHEMA: Sequence[tuple[str, Callable[[Any], str]]] = ( ("dtype", report.format_str), ("direction", report.format_str), @@ -234,3 +316,31 @@ def calculate_throughput_metrics( return { f"{prefix}_bandwidth_per_device_gb_s": bandwidth_gb_s, } + + def get_peak_link_bandwidth( + self, + ) -> tuple[float, constants.TransferLink] | None: + """Returns the D2D peak and its bounding link, or None (with a warning). + + The peak is not modeled (and the roofline columns are omitted) for: + - Non-TPU devices without chip coordinates. + - Pairs on different slices (DCN). + - Hardware specs without a bandwidth for the bounding link. + """ + if self.mesh is None: + raise ValueError("Mesh not initialized.") + # Indices refer to mesh positions, which need not follow chip order, so + # classify the actual devices rather than the indices. + src = self.mesh.devices.flat[self.config.src_device_index] + dst = self.mesh.devices.flat[self.config.dst_device_index] + try: + link = classify_transfer_link(src, dst) + peak_gb_s = device_to_device_peak_bandwidth_gb_s( + link, self.config.direction, self.hardware_spec + ) + except ValueError as e: + logging.warning( + "Skipping device_to_device *_roofline_efficiency_pct: %s.", e + ) + return None + return peak_gb_s, link diff --git a/src/accelerator_microbenchmarks/core/base.py b/src/accelerator_microbenchmarks/core/base.py index 21b5e929..fcec2904 100644 --- a/src/accelerator_microbenchmarks/core/base.py +++ b/src/accelerator_microbenchmarks/core/base.py @@ -557,6 +557,20 @@ def get_peak_ici_bandwidth_per_chip(self) -> float | None: " RooflineMode.NETWORK_ICI." ) + def get_peak_link_bandwidth( + self, + ) -> tuple[float, constants.TransferLink] | None: + """Return the peak link bandwidth per device (GB/s) and the bounding link. + + Required when roofline_mode == RooflineMode.NETWORK_LINK. Returns None + when the peak cannot be modeled for the current configuration. + """ + raise NotImplementedError( + f"{self.__class__.__name__} must implement" + " get_peak_link_bandwidth() when roofline_mode ==" + " RooflineMode.NETWORK_LINK." + ) + @abc.abstractmethod def get_total_bytes(self) -> float: """Calculate total bytes moved to/from HBM.""" diff --git a/src/accelerator_microbenchmarks/core/constants.py b/src/accelerator_microbenchmarks/core/constants.py index 262f7872..807502e1 100644 --- a/src/accelerator_microbenchmarks/core/constants.py +++ b/src/accelerator_microbenchmarks/core/constants.py @@ -42,12 +42,29 @@ class RooflineMode(str, enum.Enum): supplied by the benchmark (currently only `all_to_all`, whose peak is bounded by bisection bandwidth). Emits `peak_ici_bandwidth_per_chip_gb_s` and `_ici_roofline_efficiency_pct`. + NETWORK_LINK: Device-to-device transfer roofline: the measured per-device + bandwidth is compared against the peak of the single physical link that + carries the transfer (ICI between chips, or the die-to-die link between + the dies of one chip), as supplied by the benchmark. Emits + `peak_link_bandwidth_per_device_gb_s`, `roofline_bound_by` (a + `TransferLink` value), and `_roofline_efficiency_pct`. """ NONE = "none" COMPUTE = "compute" MEMORY_HBM = "memory_hbm" NETWORK_ICI = "network_ici" + NETWORK_LINK = "network_link" + + +class TransferLink(enum.StrEnum): + """Physical link bounding a device-to-device transfer. + + The values are the strings emitted in the `roofline_bound_by` metric. + """ + + ICI = "ici" + DIE_TO_DIE = "die_to_die" class TimingDomain(enum.StrEnum): diff --git a/src/accelerator_microbenchmarks/core/roofline.py b/src/accelerator_microbenchmarks/core/roofline.py index f69bf554..80b619df 100644 --- a/src/accelerator_microbenchmarks/core/roofline.py +++ b/src/accelerator_microbenchmarks/core/roofline.py @@ -26,6 +26,8 @@ def apply_roofline_analysis( `_memory_roofline_efficiency_pct` apply invariantly to both scopes. `_ici_roofline_efficiency_pct` is defined per chip only, since ICI collectives report exclusively `*_bandwidth_per_chip_gb_s`. + `_roofline_efficiency_pct` (NETWORK_LINK) is defined per device, + since device-to-device transfers report `*_bandwidth_per_device_gb_s`. Args: benchmark_instance: The benchmark instance being analyzed. @@ -119,4 +121,19 @@ def apply_roofline_analysis( ) * 100.0 return metrics + if mode == constants.RooflineMode.NETWORK_LINK: + peak = benchmark_instance.get_peak_link_bandwidth() + if peak is None: + return metrics + peak_bw, link = peak + metrics["peak_link_bandwidth_per_device_gb_s"] = peak_bw + metrics["roofline_bound_by"] = str(link) + for prefix in constants.TimingDomain: + actual_bw = metrics.get(f"{prefix}_bandwidth_per_device_gb_s") + if actual_bw is not None and actual_bw >= 0 and peak_bw > 0: + metrics[f"{prefix}_roofline_efficiency_pct"] = ( + actual_bw / peak_bw + ) * 100.0 + return metrics + return metrics diff --git a/src/accelerator_microbenchmarks/core/system.py b/src/accelerator_microbenchmarks/core/system.py index 55cfb3a3..95abfb23 100644 --- a/src/accelerator_microbenchmarks/core/system.py +++ b/src/accelerator_microbenchmarks/core/system.py @@ -16,6 +16,7 @@ HardwareSpec = schema.HardwareSpec TflopsSpec = schema.TflopsSpec IciSpec = schema.IciSpec +DieToDieSpec = schema.DieToDieSpec WraparoundRule = schema.WraparoundRule HbmSpec = schema.HbmSpec diff --git a/src/accelerator_microbenchmarks/specs/schema.py b/src/accelerator_microbenchmarks/specs/schema.py index b681f829..acd1e8df 100644 --- a/src/accelerator_microbenchmarks/specs/schema.py +++ b/src/accelerator_microbenchmarks/specs/schema.py @@ -106,6 +106,19 @@ class IciSpec: wraparound_rule: WraparoundRule = WraparoundRule.NONE +@dataclasses.dataclass(frozen=True) +class DieToDieSpec: + """On-package link between the compute dies of one chip (theoretical peak). + + Attributes: + bandwidth_per_direction_gb_s: Theoretical peak bandwidth between the two + dies of a chip in one direction, in decimal GB/s. The link is + full-duplex, so each direction carries this bandwidth independently. + """ + + bandwidth_per_direction_gb_s: float + + @dataclasses.dataclass(frozen=True) class HbmSpec: """High Bandwidth Memory specifications (per-chip). @@ -128,6 +141,9 @@ class HardwareSpec: devices_per_chip: int = 1 tflops: TflopsSpec | None = None ici: IciSpec | None = None + # `None` when the chip has no die-to-die link (one device per chip) or its + # bandwidth has not been verified yet. + die_to_die: DieToDieSpec | None = None hbm: HbmSpec | None = None @property diff --git a/src/accelerator_microbenchmarks/specs/tpu7x.py b/src/accelerator_microbenchmarks/specs/tpu7x.py index 476e4773..4634fbb2 100644 --- a/src/accelerator_microbenchmarks/specs/tpu7x.py +++ b/src/accelerator_microbenchmarks/specs/tpu7x.py @@ -21,5 +21,9 @@ link_bw_per_direction_gb_s=100.0, wraparound_rule=schema.WraparoundRule.ALL_DIMS_GE_4, ), + # Each chip has two chiplets connected by a die-to-die (D2D) interface + # (https://docs.cloud.google.com/tpu/docs/tpu7x#dual-chiplet_architecture). + # The D2D link is full duplex. + die_to_die=schema.DieToDieSpec(bandwidth_per_direction_gb_s=800.0), hbm=schema.HbmSpec(peak_bw_gb_s=7380.0), ) diff --git a/tests/test_device_to_device.py b/tests/test_device_to_device.py index 2ca6d449..03d1192c 100644 --- a/tests/test_device_to_device.py +++ b/tests/test_device_to_device.py @@ -1,8 +1,10 @@ """Unit tests for device_to_device.py.""" import argparse +import dataclasses import io import os +import types from unittest import mock from absl.testing import absltest @@ -10,6 +12,7 @@ from accelerator_microbenchmarks import cli from accelerator_microbenchmarks.benchmarks import device_to_device from accelerator_microbenchmarks.core import base +from accelerator_microbenchmarks.core import constants from accelerator_microbenchmarks.core import registry from accelerator_microbenchmarks.core import report from accelerator_microbenchmarks.core import system @@ -254,6 +257,10 @@ def test_run_e2e(self): self.assertNotIn("compute_roofline_efficiency_pct", result.metrics) self.assertNotIn("peak_hbm_bw_gb_s", result.metrics) self.assertNotIn("memory_roofline_efficiency_pct", result.metrics) + # CPU devices have no chip coordinates, so the link roofline is skipped. + self.assertNotIn("peak_link_bandwidth_per_device_gb_s", result.metrics) + self.assertNotIn("roofline_bound_by", result.metrics) + self.assertNotIn("wall_clock_roofline_efficiency_pct", result.metrics) def test_get_arithmetic_intensity(self): """Verify DeviceToDeviceBenchmark does not implement get_arithmetic_intensity.""" @@ -497,5 +504,133 @@ def test_generate_benchmark_report_dual_representation(self): self.assertIn("X", rep) +def _fake_device(coords=None, slice_index=0): + """Returns a stand-in for a jax.Device with the attributes D2D inspects.""" + device = types.SimpleNamespace(slice_index=slice_index) + if coords is not None: + device.coords = coords + return device + + +class ClassifyTransferLinkTest(absltest.TestCase): + """Unit tests for classify_transfer_link.""" + + def test_same_chip_is_die_to_die(self): + self.assertEqual( + device_to_device.classify_transfer_link( + _fake_device((0, 0, 0)), _fake_device([0, 0, 0]) + ), + constants.TransferLink.DIE_TO_DIE, + ) + + def test_different_chips_is_ici(self): + self.assertEqual( + device_to_device.classify_transfer_link( + _fake_device((0, 0, 0)), _fake_device((1, 1, 0)) + ), + constants.TransferLink.ICI, + ) + + def test_different_slices_raises(self): + with self.assertRaisesRegex(ValueError, "different slices"): + device_to_device.classify_transfer_link( + _fake_device((0, 0, 0), slice_index=0), + _fake_device((0, 0, 0), slice_index=1), + ) + + def test_missing_coords_raises(self): + with self.assertRaisesRegex(ValueError, "no physical chip coordinates"): + device_to_device.classify_transfer_link( + _fake_device(), _fake_device((0, 0, 0)) + ) + + +class DeviceToDevicePeakBandwidthTest(parameterized.TestCase): + """Unit tests for device_to_device_peak_bandwidth_gb_s.""" + + @parameterized.parameters( + (constants.TransferLink.ICI, "uni", 100.0), + (constants.TransferLink.ICI, "bi", 200.0), + (constants.TransferLink.DIE_TO_DIE, "uni", 800.0), + (constants.TransferLink.DIE_TO_DIE, "bi", 1600.0), + ) + def test_tpu7x_peak(self, link, direction, expected_gb_s): + self.assertEqual( + device_to_device.device_to_device_peak_bandwidth_gb_s( + link, + device_to_device.TransferDirection(direction), + system.TPU7X_HARDWARE_SPEC, + ), + expected_gb_s, + ) + + def test_missing_die_to_die_spec_raises(self): + with self.assertRaisesRegex(ValueError, "no die-to-die spec for v6e"): + device_to_device.device_to_device_peak_bandwidth_gb_s( + constants.TransferLink.DIE_TO_DIE, + device_to_device.TransferDirection.UNI, + system.V6E_HARDWARE_SPEC, + ) + + def test_missing_ici_spec_raises(self): + spec = dataclasses.replace(system.V6E_HARDWARE_SPEC, ici=None) + with self.assertRaisesRegex(ValueError, "no ICI spec for v6e"): + device_to_device.device_to_device_peak_bandwidth_gb_s( + constants.TransferLink.ICI, + device_to_device.TransferDirection.UNI, + spec, + ) + + +class GetPeakLinkBandwidthTest(parameterized.TestCase): + """Unit tests for DeviceToDeviceBenchmark.get_peak_link_bandwidth.""" + + def _make_benchmark(self, devices, src, dst, direction="uni", spec=None): + bm = device_to_device.DeviceToDeviceBenchmark( + device_to_device.DeviceToDeviceTestCaseParams( + direction=direction, src_device_index=src, dst_device_index=dst + ), + hardware_spec=spec or system.TPU7X_HARDWARE_SPEC, + ) + bm.mesh = types.SimpleNamespace(devices=types.SimpleNamespace(flat=devices)) + return bm + + def test_classifies_mesh_devices_not_indices(self): + # Mesh positions 0 and 1 hold devices on different chips, and positions + # 0 and 2 hold the two dies of one chip. + devices = [ + _fake_device((0, 0, 0)), + _fake_device((1, 0, 0)), + _fake_device((0, 0, 0)), + ] + self.assertEqual( + self._make_benchmark(devices, 0, 1).get_peak_link_bandwidth(), + (100.0, constants.TransferLink.ICI), + ) + self.assertEqual( + self._make_benchmark(devices, 0, 2, "bi").get_peak_link_bandwidth(), + (1600.0, constants.TransferLink.DIE_TO_DIE), + ) + + def test_unmodeled_pair_returns_none(self): + devices = [_fake_device((0, 0, 0)), _fake_device((0, 0, 0))] + bm = self._make_benchmark(devices, 0, 1, spec=system.V6E_HARDWARE_SPEC) + with self.assertLogs(level="WARNING") as logs: + self.assertIsNone(bm.get_peak_link_bandwidth()) + self.assertIn("no die-to-die spec", logs.output[0]) + + def test_roofline_metrics(self): + devices = [_fake_device((0, 0, 0)), _fake_device((0, 0, 0))] + bm = self._make_benchmark(devices, 0, 1) + metrics = bm.apply_roofline_analysis({ + "wall_clock_bandwidth_per_device_gb_s": 400.0, + "xprof_bandwidth_per_device_gb_s": 680.0, + }) + self.assertEqual(metrics["peak_link_bandwidth_per_device_gb_s"], 800.0) + self.assertEqual(metrics["roofline_bound_by"], "die_to_die") + self.assertAlmostEqual(metrics["wall_clock_roofline_efficiency_pct"], 50.0) + self.assertAlmostEqual(metrics["xprof_roofline_efficiency_pct"], 85.0) + + if __name__ == "__main__": absltest.main() diff --git a/tests/test_roofline.py b/tests/test_roofline.py index 4c5513f3..5548ddd5 100644 --- a/tests/test_roofline.py +++ b/tests/test_roofline.py @@ -458,6 +458,45 @@ def test_apply_roofline_analysis_network_ici_no_peak(self): res = roofline.apply_roofline_analysis(self.mock_benchmark, metrics) self.assertEqual(res, {"wall_clock_bandwidth_per_chip_gb_s": 100.0}) + def test_apply_roofline_analysis_network_link_mode(self): + """Verify NETWORK_LINK emits peak, bounding link and per-domain efficiency.""" + self.mock_benchmark.roofline_mode = constants.RooflineMode.NETWORK_LINK + self.mock_benchmark.get_peak_link_bandwidth.return_value = ( + 100.0, + constants.TransferLink.ICI, + ) + metrics = { + "wall_clock_bandwidth_per_device_gb_s": 80.0, + "xprof_bandwidth_per_device_gb_s": 93.0, + } + res = roofline.apply_roofline_analysis(self.mock_benchmark, metrics) + self.assertEqual(res["peak_link_bandwidth_per_device_gb_s"], 100.0) + self.assertEqual(res["roofline_bound_by"], "ici") + self.assertAlmostEqual(res["wall_clock_roofline_efficiency_pct"], 80.0) + self.assertAlmostEqual(res["xprof_roofline_efficiency_pct"], 93.0) + self.assertNotIn("peak_ici_bandwidth_per_chip_gb_s", res) + self.assertNotIn("wall_clock_ici_roofline_efficiency_pct", res) + + def test_apply_roofline_analysis_network_link_not_clamped(self): + """Verify efficiency above 100% is reported as-is.""" + self.mock_benchmark.roofline_mode = constants.RooflineMode.NETWORK_LINK + self.mock_benchmark.get_peak_link_bandwidth.return_value = ( + 100.0, + constants.TransferLink.ICI, + ) + metrics = {"wall_clock_bandwidth_per_device_gb_s": 120.0} + res = roofline.apply_roofline_analysis(self.mock_benchmark, metrics) + self.assertAlmostEqual(res["wall_clock_roofline_efficiency_pct"], 120.0) + self.assertNotIn("xprof_roofline_efficiency_pct", res) + + def test_apply_roofline_analysis_network_link_no_peak(self): + """Verify no link roofline keys are added without a modeled peak.""" + self.mock_benchmark.roofline_mode = constants.RooflineMode.NETWORK_LINK + self.mock_benchmark.get_peak_link_bandwidth.return_value = None + metrics = {"wall_clock_bandwidth_per_device_gb_s": 80.0} + res = roofline.apply_roofline_analysis(self.mock_benchmark, metrics) + self.assertEqual(res, {"wall_clock_bandwidth_per_device_gb_s": 80.0}) + if __name__ == "__main__": absltest.main() diff --git a/tests/test_system.py b/tests/test_system.py index 1eb80b22..b8f28e62 100644 --- a/tests/test_system.py +++ b/tests/test_system.py @@ -194,6 +194,9 @@ def test_hardware_spec_tpu7x_presets(self): hw_spec.ici.wraparound_rule, system.WraparoundRule.ALL_DIMS_GE_4 ) + # Test die-to-die stats + self.assertEqual(hw_spec.die_to_die.bandwidth_per_direction_gb_s, 800.0) + # Test HBM stats self.assertEqual(hw_spec.hbm.peak_bw_gb_s, 7380.0) self.assertEqual(hw_spec.peak_hbm_bandwidth_per_device, 3690.0) @@ -225,6 +228,9 @@ def test_hardware_spec_v6e_presets(self): hw_spec.ici.wraparound_rule, system.WraparoundRule.PER_DIM_GE_16 ) + # One device per chip: no die-to-die link. + self.assertIsNone(hw_spec.die_to_die) + # Test HBM stats self.assertEqual(hw_spec.hbm.peak_bw_gb_s, 1638.4) self.assertEqual(hw_spec.peak_hbm_bandwidth_per_device, 1638.4)