Add a benchmark harness and record the performance baseline - #95
Merged
Conversation
There was no way to tell whether a change made the library faster or slower, which makes any optimisation unfalsifiable. This adds the measurement side of the safety net that the regression goldens started. benchmarks/bench_engines.py sweeps seven configurations spanning single-mode, mixed-state, polychrome, multislice, a heavy combined case and OPR. It reports wall time *and* peak GPU memory, because the group runs cards as small as 32 GB and an optimisation that trades memory for speed is a regression here, not an improvement. Datasets are synthesised in-process at a requested size, so it runs anywhere without the gitignored contents of example_data/. It is headless: DummyMonitor, matplotlib Agg, TQDM_DISABLE set before tqdm is imported, since mPIE draws a progress bar per scan position and that lands inside the loop being measured. Each configuration gets a warm-up iteration first so cuFFT plans and cached transfer functions are not attributed to the measurement. No runtime code is touched. BASELINE.md records the numbers at 0.2.8, and two of them are worth stating: - Per-position cost is flat at 586-749 us across a 40x range of field size (0.1 -> 4.0 MB). The cost barely depends on how much data is processed, so the GPU is not what is being waited on -- it is host-side dispatch. - Two thirds of an OPR iteration is linear algebra rather than ptychography, and orthogonalizeIncoherentModes (39.6%) is the larger half, not orthogonalizeProbeStack (26.5%). It also records two measured negative results so nobody repeats them: padding FFT sizes is slower, not faster (364->384 measured 0.93x), and batching the existing per-frame SVD buys 1.02x because the cost is inside the SVD rather than in launch overhead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fifth in the series. Follows #90, #91, #93, #94.
No runtime code is touched. Adds
benchmarks/only.Why
There is currently no way to tell whether a change made the library faster or slower. That makes
any optimisation unfalsifiable. The regression goldens (#90) pin what the library computes;
this pins how fast.
The harness
benchmarks/bench_engines.pysweeps seven configurations — single-mode, mixed-state,polychrome, multislice, a heavy combined case, and OPR.
It reports wall time and peak GPU memory. Both are acceptance criteria: the group runs cards
as small as 32 GB, so an optimisation that trades memory for speed is a regression here, not an
improvement.
Details that matter for the numbers being trustworthy:
example_data/DummyMonitor, matplotlibAgg, andTQDM_DISABLEset before tqdm isimported, since mPIE draws a progress bar per scan position and that lands inside the loop
being measured
attributed to the measurement
Baseline at 0.2.8
Two findings worth pulling out.
Per-position cost is flat across a 40x range of field size — 586 to 749 µs from 0.1 MB to
4.0 MB. The cost barely depends on how much data is being processed, which means the GPU is not
what is being waited on. It is host-side dispatch.
Two thirds of an OPR iteration is linear algebra, not ptychography:
orthogonalizeIncoherentModesorthogonalizeProbeStackorthogonalizeIncoherentModes— a Python loop running one small SVD per frame — is the largerhalf. That was not obvious beforehand.
Recorded negative results
So nobody spends a week re-deriving them:
2²·7·13) looks like an awkward cuFFTsize but runs 2.6 µs/MB against 1.9 best-case. Padding 364→384 measured 0.93x; 364→512 was
0.65x.
so batching the same algorithm does not help. Only changing the algorithm does.
Note on scope
BASELINE.mddeliberately contains only what is true onmaintoday. The Gram-orthogonalisationmeasurements and results belong with the change that implements them, and land in that PR rather
than here.
No version bump — nothing ships to PyPI from this.