Skip to content

Add a benchmark harness and record the performance baseline - #95

Merged
ShantanuKodgirwar merged 1 commit into
mainfrom
performance-benchmarks
Aug 3, 2026
Merged

Add a benchmark harness and record the performance baseline#95
ShantanuKodgirwar merged 1 commit into
mainfrom
performance-benchmarks

Conversation

@ShantanuKodgirwar

Copy link
Copy Markdown
Contributor

Fifth in the series. Follows #90, #91, #93, #94.

No runtime code is touched. Adds benchmarks/ only.

Why

There is currently no way to tell whether a change made the library faster or slower. That makes
any optimisation unfalsifiable. The regression goldens (#90) pin what the library computes;
this pins how fast.

The harness

benchmarks/bench_engines.py sweeps seven configurations — single-mode, mixed-state,
polychrome, multislice, a heavy combined case, and OPR.

python benchmarks/bench_engines.py --markdown   # BASELINE.md table
python benchmarks/bench_engines.py --quick      # one config per regime
python benchmarks/bench_engines.py --cpu        # CPU baseline too

It reports wall time and peak GPU memory. Both are acceptance criteria: the group runs cards
as small as 32 GB, so an optimisation that trades memory for speed is a regression here, not an
improvement.

Details that matter for the numbers being trustworthy:

  • datasets are synthesised in-process, so it runs anywhere without the gitignored contents of
    example_data/
  • headlessDummyMonitor, matplotlib Agg, and TQDM_DISABLE set before tqdm is
    imported, since mPIE draws a progress bar per scan position and that lands inside the loop
    being measured
  • a warm-up iteration per config, so cuFFT plans and cached transfer functions are not
    attributed to the measurement

Baseline at 0.2.8

config reg field MB ms/iter µs/pos peak GB
simu-like 128, 100fr A 0.1 58.6 586 0.03
USAF-like 364, 102fr A 1.0 71.4 700 0.23
mixed npsm=4 364 A 4.0 72.5 711 0.26
Brain-like poly=7 182 A 1.8 74.9 749 0.06
multislice nslice=4 364 A 4.0 150.6 1477 0.26
heavy 7×2×4 364 B 56.6 90.0 2249 1.02
OPR 364, 202fr, 4 modes C 4.0 505.4 2502 2.85

Two findings worth pulling out.

Per-position cost is flat across a 40x range of field size — 586 to 749 µs from 0.1 MB to
4.0 MB. The cost barely depends on how much data is being processed, which means the GPU is not
what is being waited on. It is host-side dispatch.

Two thirds of an OPR iteration is linear algebra, not ptychography:

stage ms/iter share
orthogonalizeIncoherentModes 199.9 39.6%
orthogonalizeProbeStack 133.8 26.5%
position loop and everything else 170.8 33.9%

orthogonalizeIncoherentModes — a Python loop running one small SVD per frame — is the larger
half. That was not obvious beforehand.

Recorded negative results

So nobody spends a week re-deriving them:

  • FFT-size padding is slower, not faster. Np=364 (2²·7·13) looks like an awkward cuFFT
    size but runs 2.6 µs/MB against 1.9 best-case. Padding 364→384 measured 0.93x; 364→512 was
    0.65x.
  • Batching the existing SVD buys 1.02x. The cost is inside the SVD, not in launch overhead,
    so batching the same algorithm does not help. Only changing the algorithm does.

Note on scope

BASELINE.md deliberately contains only what is true on main today. The Gram-orthogonalisation
measurements and results belong with the change that implements them, and land in that PR rather
than here.

No version bump — nothing ships to PyPI from this.

There was no way to tell whether a change made the library faster or slower,
which makes any optimisation unfalsifiable. This adds the measurement side of
the safety net that the regression goldens started.

benchmarks/bench_engines.py sweeps seven configurations spanning single-mode,
mixed-state, polychrome, multislice, a heavy combined case and OPR. It reports
wall time *and* peak GPU memory, because the group runs cards as small as
32 GB and an optimisation that trades memory for speed is a regression here,
not an improvement.

Datasets are synthesised in-process at a requested size, so it runs anywhere
without the gitignored contents of example_data/. It is headless: DummyMonitor,
matplotlib Agg, TQDM_DISABLE set before tqdm is imported, since mPIE draws a
progress bar per scan position and that lands inside the loop being measured.
Each configuration gets a warm-up iteration first so cuFFT plans and cached
transfer functions are not attributed to the measurement.

No runtime code is touched.

BASELINE.md records the numbers at 0.2.8, and two of them are worth stating:

- Per-position cost is flat at 586-749 us across a 40x range of field size
  (0.1 -> 4.0 MB). The cost barely depends on how much data is processed, so
  the GPU is not what is being waited on -- it is host-side dispatch.
- Two thirds of an OPR iteration is linear algebra rather than ptychography,
  and orthogonalizeIncoherentModes (39.6%) is the larger half, not
  orthogonalizeProbeStack (26.5%).

It also records two measured negative results so nobody repeats them: padding
FFT sizes is slower, not faster (364->384 measured 0.93x), and batching the
existing per-frame SVD buys 1.02x because the cost is inside the SVD rather
than in launch overhead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ShantanuKodgirwar
ShantanuKodgirwar merged commit 4e99c8a into main Aug 3, 2026
3 checks passed
@ShantanuKodgirwar
ShantanuKodgirwar deleted the performance-benchmarks branch August 3, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant