bench: cross-library Pauli-propagation benchmark, with truncation-accuracy measurement - #211
Open
Roger-luo wants to merge 5 commits into
Open
bench: cross-library Pauli-propagation benchmark, with truncation-accuracy measurement#211Roger-luo wants to merge 5 commits into
Roger-luo wants to merge 5 commits into
Conversation
…on engines Adds a single-threaded harness that runs the same two Trotter workloads — TFIM magnetization and a Heisenberg autocorrelator — through ppvm, PauliPropagation.jl, PauliStrings.jl, Qiskit's pauli-prop and Algorithmiq's monoprop, from one shared environment contract into one CSV. The driver refuses to report a timing until every engine has dumped its full propagated support at a small width and matched ppvm's term-for-term. That check is load-bearing rather than decorative: it is what pinned down monoprop's exp(+iθH) angle convention (θ = -θ_spec/2 on a reversed gate list), which otherwise reproduces the observable to nine digits off the wrong operator. monoprop needs a second guard. Left alone it takes one serial partition per physical core, so an uncapped run reports a ~3x faster wall time than the serial one against engines that never had the option. The runner pins monoprop_NUM_THREADS=1 / monoprop_PARTITIONS=off, and because both are read once into a cached C++ static, it also measures its own CPU/wall ratio around the timed region and exits non-zero if the cap did not land. On an M4 Pro, ppvm beats both Julia engines everywhere (2.2-11.3x vs PauliPropagation.jl, 4.9-13.2x vs PauliStrings.jl, on an identical support) and loses to monoprop on both models, by up to 3.0x on TFIM at n=64. The README records the run along with the per-engine truncation differences that a ratio should not be read without. Co-authored-by: Cursor <cursoragent@cursor.com>
The README described the two truncation policies and their support difference but not what the difference costs. Adds the coefficient-vector L2 error against a converged reference, which puts monoprop within 12% of ppvm and pauli-prop at 1.7-2.0x, and records that the scalar observable is the wrong yardstick here because its truncation error changes sign. Co-authored-by: Cursor <cursoragent@cursor.com>
The README quoted error ratios with no data behind them and no way to regenerate them. Adds xbench_accuracy.py, which diffs each engine's whole coefficient vector against a converged reference across atol and writes both the vector norms and the scalar observable error, and commits the run as accuracy.csv. Extending the sweep from three engines to all five corrects the pauli-prop range in the README, 1.23-2.60x rather than 1.7-2.0x, and shows both Julia engines reproducing ppvm's error columns exactly -- independent confirmation that the three accumulate-then-truncate engines share one truncation rule. Co-authored-by: Cursor <cursoragent@cursor.com>
…ation rules diverge The two Trotter models are a weak test bed for truncation questions: they are nearest-neighbour, uniform-angle, and conserve enough structure that their support saturates a symmetry sector rather than exploring the space. On them the prune-at-creation and accumulate-then-truncate rules stay within 1.2x of each other, which says more about the circuits than about the rules. Adds `MODEL=scramble` to the ppvm and monoprop runners: steps*n random all-to-all two-qubit Pauli rotations with random axes and random angles, built from a splitmix64 stream reimplemented identically in Rust and Python so both engines propagate the same circuit. Validated term-for-term across three seeds/widths/angles, and the existing tfim/heisenberg agreement across all five engines is unaffected. With that instance the rules do separate, and only when three conditions hold at once -- small sin(theta) to widen the rejection band, depth to populate and compound it, and a scrambled operator so the loss does not average out. At n=8, 3200 gates, angles under 0.1 and atol=1e-4, monoprop's coefficient-vector error is 2.96x ppvm's, tight at 2.90-3.06x over five seeds. Records the sweep plus a theta=pi/4 control that reverses the sign (0.85-0.96x, monoprop ahead), and documents the mechanism: merge multiplicity is exactly two, which caps the divergence and ties both errors to atol. Co-authored-by: Cursor <cursoragent@cursor.com>
…s sweeps The error-scaling story was spread across ad-hoc runs whose output only ever reached a terminal, so none of the ratios quoted in the README could be checked or replotted. Generalises xbench_accuracy.py to sweep models, depth, angle, seeds and atol as comma-separated lists over one flattened product, and to record ||c*|| so the absolute errors can be read as relative ones. Adds plot_accuracy.py: four panels, one per question the data was asked. Consolidates the divergence sweeps (depth, angle, scrambler, and the theta=pi/4 control) into accuracy_divergence.csv, replacing accuracy_scramble.csv. The panels are chosen so the two easy misreadings are hard to make: panel 1 shows every engine falling one decade per decade of atol, so the rules share a convergence order, and panel 3 shows the ratio crossing 1.0 with angle, so no single ratio characterises them. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a cross-library Pauli-propagation benchmark harness to compare ppvm (Rust) against Julia (PauliPropagation.jl, PauliStrings.jl) and Python bindings (Qiskit pauli-prop, Algorithmiq monoprop), including a validation gate (term-for-term support diff) and an accuracy-vs-truncation sweep/plots to make runtime comparisons meaningful across differing truncation semantics.
Changes:
- Introduces per-engine runners plus a unified driver (
run_xbench.py) that validates identical propagated support before timing and emits a merged CSV. - Adds an accuracy measurement pipeline (
xbench_accuracy.py,plot_accuracy.py, checked-in CSVs) to quantify truncation error alongside runtime. - Adds Julia bench scripts and updates benchmark docs/metadata to include PauliStrings.jl.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| julia-benchmarks/Project.toml | Adds PauliStrings.jl dependency for the Julia side of the benchmark. |
| julia-benchmarks/Manifest.toml | Updates Julia manifest to include PauliStrings.jl and its dependencies / version bumps. |
| julia-benchmarks/benches/xbench_ps.jl | New PauliStrings.jl runner implementing the shared environment/CSV contract and dump format. |
| julia-benchmarks/benches/xbench_pp.jl | New PauliPropagation.jl runner implementing the shared environment/CSV contract and dump format. |
| crates/ppvm-pauli-sum/examples/xbench.rs | New ppvm runner (Rust) for the shared workloads, including the scramble workload and dump output. |
| benchmarks/README.md | Documents the new cross-library benchmark and links out to the detailed README. |
| benchmarks/cross-library/xbench_qiskit.py | New Qiskit pauli-prop runner with max-terms guard and support-dump normalization. |
| benchmarks/cross-library/xbench_monoprop.py | New monoprop runner with thread/partition pinning and CPU/wall sanity check. |
| benchmarks/cross-library/xbench_accuracy.py | New tool to sweep atol and compare full coefficient vectors to a converged reference. |
| benchmarks/cross-library/run_xbench.py | New driver: invokes runners, validates dumps, merges results, prints summary table. |
| benchmarks/cross-library/README.md | Full spec/contract, rationale, caveats, and example commands/results for the harness. |
| benchmarks/cross-library/plot_xbench.py | New plotting script for runtime/relative runtime/workload panels from results.csv. |
| benchmarks/cross-library/plot_accuracy.py | New plotting script rendering truncation-error scaling and divergence panels. |
| benchmarks/cross-library/accuracy.csv | Checked-in accuracy sweep output (5 engines vs atol) used for plots/docs. |
| benchmarks/cross-library/accuracy_divergence.csv | Checked-in divergence sweeps (depth/angle/scramble) used for plots/docs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+95
to
+96
| /// The `scramble` workload: `steps · n` two-qubit Pauli rotations on uniformly | ||
| /// random *all-to-all* pairs, with random axes and random angles in `(0, 2·J·dt]`. |
Comment on lines
+80
to
+82
| `steps·n` two-qubit Pauli rotations `exp(−iθ/2·Pₐ⊗P_b)`, each drawing a uniformly | ||
| random **all-to-all** pair `a≠b`, random axes `Pₐ, P_b ∈ {X,Y,Z}` and a random | ||
| angle `θ ∈ (0, 2J·dt]`. Observable and readout as for `heisenberg`. |
Comment on lines
+209
to
+214
| for dt, dash, tag in ( | ||
| ("0.05", "-", "dt=0.05, 3200 gates"), | ||
| ("0.7853981633974483", "--", "dt=π/4, 80 gates"), | ||
| ): | ||
| sub = [r for r in rows if r["dt"] == dt] | ||
| for lib in PAIR: |
Comment on lines
+169
to
+185
| print("validating: all engines must agree term-for-term", file=sys.stderr) | ||
| for model in ("tfim", "heisenberg"): | ||
| env = dict(params) | ||
| env.update( | ||
| {"MODEL": model, "QUBITS": "4", "STEPS": "3", "ATOL": "1e-14", "DUMP": "1"} | ||
| ) | ||
| reference: dict[str, float] | None = None | ||
| ref_lib = "" | ||
| for lib in libs: | ||
| terms = parse_dump(invoke(RUNNERS[lib], env, quiet=True)) | ||
| if reference is None: | ||
| reference, ref_lib = terms, lib | ||
| print( | ||
| f" {model}: {lib} — {len(terms)} terms (reference)", | ||
| file=sys.stderr, | ||
| ) | ||
| continue |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmarks
ppvmagainst the four other single-threaded Pauli-propagation engines we know of — PauliPropagation.jl, PauliStrings.jl, pauli-prop (Qiskit) and monoprop (Algorithmiq) — on two Trotter workloads plus a random scrambling circuit.Validation before timing. No number is reported until every engine's propagated support matches
ppvm's term-for-term. This is the load-bearing part: every bug it caught while the harness was being written produced plausible numbers that agreed on the scalar observable to nine digits — a reversed circuit in thepauli-proprunner, duplicate Paulis collapsing in its readout, and monoprop'sexp(+iθH)sign convention.Accuracy is measured alongside runtime, because the engines don't all mean the same thing by
atol.ppvm, PauliPropagation.jl and PauliStrings.jl accumulate every contribution to a term and threshold the merged coefficient;pauli-propand monoprop threshold each branch as it is emitted. Same knob, different rule, so runtime alone isn't comparable.xbench_accuracy.pydiffs each engine's whole coefficient vector against a converged reference andplot_accuracy.pyrenders the scaling.A
scrambleworkload (steps·nrandom all-to-all two-qubit Pauli rotations, random axes and angles) was added because the two Trotter models are a weak test bed for truncation questions — they're nearest-neighbour, uniform-angle, and their support saturates a symmetry sector rather than exploring the space. Both runners generate it from a splitmix64 stream reimplemented identically in Rust and Python; the term-for-term diff enforces that they stay in sync.What the measurements say
atol, so the two truncation rules share a convergence order and differ only in a constant: monoprop 1.00–1.12× ofppvm's error,pauli-prop1.23–2.60×. Both Julia engines reproduceppvm's error columns to every digit, which independently confirms they implement the same rule.ppvm's rule has its own failure mode —truncate()after each gate permanently deletes a term whose merged coefficient transiently cancelled. No single ratio characterises the two rules.sin θ, large depth, and a scrambled operator hold simultaneously. The structural reason they can't separate further: a Pauli rotation sendsQto exactly two words, so a merged coefficient can exceed its larger contribution by at most 2×, which anchors both engines' losses to the sameatol.Caveats that affect how the ratios should be read — differing supports, monoprop's zero-coefficient rows and mandatory weight
cutoff,pauli-prop's mandatorymax_terms— are documented under "Known differences between the engines" in the README, and the plot carries the workload size beside the runtime so a ratio is never read without it.Test plan
cargo test --workspacecargo clippy/cargo fmt/ruff/tyclean (pre-commit hooks pass)scramblevalidated betweenppvmand monoprop across three seeds, widths and anglesNote:
scrambleis implemented only in theppvmand monoprop runners (--libs ppvm,monoprop); the other three are untouched and keep working ontfim/heisenberg.Made with Cursor