traits-2: rebuild the trait tower, engine, tableau and symbolic ring as ppvm-*-2

[Roger-luo]

Rebuilds the trait system, propagation engine, stabilizer tableau and symbolic
coefficient ring as a parallel ppvm-*-2 family, then routes every downstream
consumer to it behind a feature flag. Old crates untouched, still the default —
no rename, no deletion.

The stabilizer tableau was then rebuilt a second time, after the first
audit: four square quadrants in one contiguous qubit-contiguous (column-major)
allocation, with an inverse tableau read out of the same arena — Stim's
Tableau / inv_state layout. That is a deliberate trade, and the numbers below
show both sides of it. A Clifford gate touches two columns instead of all 2n
rows (4.9× faster at n=128); a deterministic measurement is one sign read
(surface_d30, 1889 qubits, is 15.9× faster than legacy); and a row fold on
a dense frame now pays a transpose the row-major layout got for free (the
85-qubit MSD sweep is 2.7× slower batched, 7.8× slower measured one qubit
at a time). See "Tableau layout: what it bought, what it cost".

flowchart LR
  T["ppvm-traits-2<br/>traits · leaf types<br/>blanket Clifford"]
  W["ppvm-pauli-word-2<br/>packed PauliWord"]
  L["ppvm-lossy-pauli-word-2<br/>+ loss plane"]
  P["ppvm-phased-pauli-word-2<br/>Phased·W · fused Clifford"]
  S["ppvm-pauli-sum-2<br/>Sum · policies · L4<br/>ColumnStore · IndexMapStore"]
  B["ppvm-tableau-2<br/>Tableau · Generalized<br/>· mixture"]
  Y["ppvm-sym-2<br/>exact / symbolic ring"]
  D["downstream<br/>stim · vihaco · cli · tui<br/>legacy default | traits-2 opt-in"]
  T --> W
  W --> L
  W --> P
  T --> S
  W --> S
  L --> S
  T --> B
  W --> B
  S --> B
  T --> Y
  S --> Y
  S --> D
  B --> D

Loading

ppvm_traits::Config bundled six unrelated axes into one trait, so varying any
one of them meant rewriting an impl Config wholesale. It decomposes into:

flowchart LR
  CFG["Config<br/>one impl per<br/>combination"]
  CFG --> A["Coefficient<br/>ring + magnitude only<br/>(admits exact rings)"]
  CFG --> B["Angle · Halvable<br/>ImaginaryUnit · Conjugate<br/>opt-in capabilities"]
  CFG --> C["Word · PauliBits<br/>Indexable<br/>read / bits / key identity"]
  CFG --> D["Support · Accumulate · Scale<br/>Pair · Multiply<br/>graded layers on the container"]
  CFG --> E["Policy<br/>capacity + truncation"]
  CFG --> F["key_hash<br/>pass-through hasher<br/>(map picks none)"]

Loading

Results — headline workloads

The circuits we actually care about. Measured for this PR, not pulled from the
audit: the propagation rows at 01392466, every tableau row re-measured at
43d6a816 after the column-major rebuild.

PAULI PROPAGATION — Trotter propagation, 10 steps, truncated at 1e-6
same gate list / order / truncation on both engines; supports diffed term-for-term

TFIM      H = J.SUM(Zi Zi+1) + h.SUM(Xi),  observable SUM(Zi)
          RX layer then RZZ layer per step

      n        terms      ppvm-1      ppvm-2   new/old
      8        4,701      3.4 ms      3.0 ms    0.882x  █████████████████████░░░░░
     16       19,529     12.2 ms     10.5 ms    0.861x  ████████████████████░░░░░░
     24       34,353     23.3 ms     21.2 ms    0.910x  ██████████████████████░░░░
     32       49,177     39.6 ms     34.7 ms    0.876x  █████████████████████░░░░░
     40       64,001     60.7 ms     54.3 ms    0.895x  █████████████████████░░░░░
     48       78,825     84.9 ms     74.1 ms    0.873x  █████████████████████░░░░░

HEISENBERG   H = J.SUM(XX+YY+ZZ) + h.SUM(Zi),  autocorrelator
             the branch-heavy model: support grows ~40x faster in n

      n        terms      ppvm-1      ppvm-2   new/old
      6        1,022      2.8 ms      2.8 ms    1.000x  ████████████████████████░░
      8       16,324     46.2 ms     39.3 ms    0.851x  ████████████████████░░░░░░
     10      225,353    465.1 ms    375.4 ms    0.807x  ███████████████████░░░░░░░
     12    1,174,849      2.52 s      1.95 s    0.774x  ██████████████████░░░░░░░░
     14    2,915,879      5.54 s      4.32 s    0.780x  ██████████████████░░░░░░░░

Both propagation rows come from mise run xbench --libs ppvm-1,ppvm-2, which
refuses to report a timing until it has diffed the two engines' propagated
supports term for term — the observable agrees to 10 digits at every width.
Heisenberg is the interesting one: its support grows far faster in n, and the
-2 advantage grows with it, from parity at n=6 to 1.28× faster at n=14
(2.9 M terms).

TABLEAU — Clifford + measurement size sweep, n = 32..128, 3 launches
what the column-major layout is for: one qubit, all generators

                            ppvm-tableau         -2   new/old          range
  Clifford circuit, n=32       2.40 us    2.04 us    0.847x  0.836-0.850  ████████████████████░░░░
  Clifford circuit, n=64       8.60 us    3.40 us    0.421x  0.395-0.426  ██████████░░░░░░░░░░░░░░
  Clifford circuit, n=96      17.12 us    4.91 us    0.286x  0.280-0.287  ███████░░░░░░░░░░░░░░░░░
  Clifford circuit, n=128     29.73 us    6.06 us    0.202x  0.198-0.205  █████░░░░░░░░░░░░░░░░░░░

  measure sweep, n=32          2.52 us    3.19 us    1.251x  1.240-1.265  << SLOWER
  measure sweep, n=64          9.36 us    6.40 us    0.690x  0.684-0.709  █████████████████░░░░░░░
  measure sweep, n=96         19.85 us   10.34 us    0.526x  0.494-0.531  █████████████░░░░░░░░░░░
  measure sweep, n=128        34.52 us   15.11 us    0.438x  0.434-0.441  ███████████░░░░░░░░░░░░░

TABLEAU — 1889-qubit surface-code memory, via ppvm-stim, 3 launches
27870 measurements per shot, 97% of them deterministic

                                legacy   traits-2   new/old          range
  surface_d30, per shot       623.5 ms    39.2 ms    0.063x  0.063-0.063  █░░░░░░░░░░░░░░░░░░░░░░░

TABLEAU — magic state distillation, 85 qubits (5 blocks x 17), 3 launches
the same layout's worst case: one generator, all qubits, on a dense frame

                            ppvm-tableau         -2   new/old          range
  MSD circuit, naive           98.3 us   208.3 us    2.122x  2.11-2.13    << SLOWER
  MSD circuit, fused T         60.3 us    82.4 us    1.365x  1.35-1.38    << SLOWER
  measure_all on MSD state     25.2 us    67.7 us    2.682x  2.67-2.68    << SLOWER
  measure loop on MSD state    25.1 us   197.1 us    7.826x  7.68-7.84    << SLOWER
  noisy shots                 497.5 us   893.1 us    1.795x  1.77-1.80    << SLOWER
  16 T-gates on 85q            49.5 ms    49.8 ms    1.006x  1.000-1.012  ███████████████████████░

TABLEAU — whole STIM programs, via ppvm-stim, 3 launches per backend
cross-build (see below): each row is two separately compiled binaries

                                legacy   traits-2   new/old          range
  cultivation_d5              498.9 us   433.1 us    0.880x  0.858-0.909  █████████████████████░░░
  repetition_code_d3_r3        0.63 us    1.47 us    2.340x  2.30-2.34    << SLOWER
  ghz (4q)                     0.29 us    0.81 us    2.790x  2.76-2.80    << SLOWER
  bell_pair (2q)               0.18 us    0.55 us    2.955x  2.92-3.14    << SLOWER
  feedback_cx                  0.17 us    0.48 us    2.847x  2.83-2.91    << SLOWER

Read honestly, that is one story told four times: the layout is optimal for
"one qubit, all generators" and pessimal for "one generator, all qubits" on a
frame dense enough that gathering the generators costs more than re-orienting
the whole tableau. Which side a workload lands on is decided by its measurement
mix, not its size — surface_d30 is 1889 qubits and 15.9× faster because 97% of
its measurements are deterministic; MSD is 85 qubits and 2.7× slower because its
frame is dense and every measurement projects. How the sweep is called is worth
a factor of three on top of that: batched under one orientation change it is
2.7×, one qubit at a time it is 7.8×, and the two rows are the same measurements
on the same state. The four small STIM programs are
slower for a third reason again — a fixed ~0.4 µs on the first random
measurement — which is 3× on a 2-qubit circuit and invisible past n ≈ 32.

Two provenance caveats, unchanged from before: the MSD circuit is fixed at 85
qubits, so the size sweep is a random Clifford + measurement circuit rather than
MSD at several distillation sizes; and everything routed through ppvm-stim
(surface_d30, cultivation_d5, the four small programs) compiles exactly one
backend per binary, so those rows are cross-build comparisons carrying layout
bias, unlike the paired rows above them.

Tableau layout: what it bought, what it cost

The first -2 tableau kept the old crate's row-major frame, which is why the
original audit showed it at parity on gates and slightly ahead on measurement.
It was then rebuilt on Stim's layout, in four steps, each with its own commit
and its own measurement.

Contiguous column-major storage. Four square quadrants (destabilizer/
stabilizer × X/Z) in one 32-byte-aligned allocation, indexed qubit-contiguously.
A Clifford gate is then a sweep over two majors instead of over all 2n
generators: 218 ns → 23 ns per CNOT at n=128. Because a rectangular bit-matrix
transpose is not an in-place permutation and a square one is, the quadrants are
square — the same reason Stim splits them.

Row-oriented reads gather or transpose, whichever is cheaper. A
generator-contiguous pass either materializes the k generators it needs out of
the column-major arena, or takes a TransposedTableau guard (Stim's
TableauTransposedRaii). blocks::prefer_gather picks. That took surface_d30
from 29 s/shot — the low point of the rewrite — to 224 ms.

The inverse tableau. Forward columns are inverse rows, so no second bit
matrix is needed: x-bit j of U†X_qU is the Z bit of stabilizer j at qubit
q. Only the 2n signs are tracked separately, maintained by prepending each
gate's G† table. That makes the determinism check one sign bit and
compute_decomposition's ℤ/4 fold one sign bit plus a popcount, and it
survives a projection (the elimination is a sequence of Cliffords appended to
the frame, so it conjugates each inverse row by V†). surface_d30 590 → 37 ms.

The transpose is sized to the frame. transpose_block used to run six
shift-mask rounds over all 64 rows however few were populated, so a 2-qubit frame
paid a 64-qubit orientation change. Truncating to e.next_power_of_two() rows is
exact, not a heuristic — a round j ≥ span is the identity on a corner-populated
block. First random measurement on a GHZ chain at n = 2/4/8/16: 1.33/1.34/1.38/
1.39 µs → 541/604/630/781 ns.

One orientation change per sweep, not per qubit. A dense frame that
re-orients on every measurement pays the transpose n times over. The library's
measure_all already held a single guard across the batch, but the conformance
driver's did not — its loop re-drew the legacy compatibility RNG stream per
measurement, which the batched entry points cannot express, so it stayed a
per-qubit loop and measure_all/new was silently the same benchmark as
measure_loop/new. Hoisting just the guard (GeneralizedTableau::with_row_major,
now public for exactly this kind of caller) keeps the stream and drops the MSD
sweep from 201.6 µs to 67.7 µs — level with measure_all_with_scratch. The fused
MSD circuit, which ends in measure_all, went 3.41× → 1.37×.

What it cost, and what is open. A row fold on a dense frame is the one
access pattern this layout cannot serve cheaply, and the 85-qubit MSD state is
exactly that: 0.80 µs per measurement batched, 2.4 µs unbatched, against the
row-major engine's 0.31 µs either way. Attributed with samply,
transpose_quadrants is 69% of the unbatched sweep. Two things are known about
what is left and neither is in this PR:

• A caller that measures one qubit at a time on a dense frame still pays a
  transpose pair per measurement (7.8× on the MSD loop). Holding the frame
  row-major lazily — until a column-major access actually demands it — would
  amortize that without the caller having to ask.
• The batched 2.7× wants a column-major prefix-parity scan for the inverse
  sign update, so a dense projection never re-orients at all.
• On small frames what is left is the allocator, not the transpose: 47% of a
  2-qubit case-a measurement is inside libsystem_malloc, from ~10 Vecs per
  projection that project_inverse already knows how to carve out of a reusable
  arena.

Full derivation, cost models and open questions in
docs/design/tableau-data-structure.md.

Results — full sweep, by engine

Grouped by engine, since the five engines are affected very differently.

828 old/new benchmark pairs, re-audit at e3a37026
(clifford/cy re-measured at 979efe99, after the fix)

  ████████████████████████████████████████▓▓▓▓▓▓▓▓▓▓▓░░░░░
   590 improved (<0.97x)     170 parity     66 above gate / 0 actionable

                                               new/old   faster <<<                 1.03 gate

PAULI PROPAGATION   ppvm-pauli-sum-2
  noise batch, Z errors                         0.755x   ███████████████████████░░░░░░░░┃░░░░
  clifford batch, CNOT                          0.853x   ██████████████████████████░░░░░┃░░░░
  re-key CNOT sweep                             0.871x   ██████████████████████████░░░░░┃░░░░
  noisy TFIM Trotter, n=12   [0.86-0.90]        0.880x   ██████████████████████████░░░░░┃░░░░
  native RZZ sweep                              0.960x   █████████████████████████████░░┃░░░░
  Trotter, decomposed RZZ   [0.949-0.971]       0.963x   █████████████████████████████░░┃░░░░
  qubit sweep, n=12   [0.997-1.039]             1.025x   ███████████████████████████████┃░░░░
  clifford/cy single gate   [was 1.111x]        0.895x   ████████████████████████████░░░┃░░░░

    ...same engine, storage swapped
  IndexMapStore build                           0.570x   █████████████████░░░░░░░░░░░░░░┃░░░░
  lossy interleaved GHZ n=12   [0.84 / 0.99]    0.840x   █████████████████████████░░░░░░┃░░░░
  ColumnStore rzz / hash   [0.93-0.95]          0.950x   ████████████████████████████░░░┃░░░░
  ColumnStore rx / hash   [0.94-0.97]           0.970x   █████████████████████████████░░┃░░░░
  ColumnStore Trotter / hash   [0.98-0.99]      0.990x   ██████████████████████████████░┃░░░░

WORD LAYER   PauliWord / Lossy / Phased
  pattern contains, lossy                       0.011x   █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃░░░░
  pattern contains, ordinary                    0.015x   █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃░░░░
  pattern parse, indexed                        0.478x   ██████████████░░░░░░░░░░░░░░░░░┃░░░░
  ordinary sqrt-X   [0.850-0.893]               0.870x   ██████████████████████████░░░░░┃░░░░
  phased CZ                                     0.886x   ███████████████████████████░░░░┃░░░░
  phased CNOT                                   0.980x   █████████████████████████████░░┃░░░░
  phased ZCX                                    0.989x   ██████████████████████████████░┃░░░░
  ordinary CY   [0.998-1.021]                   1.004x   ██████████████████████████████░┃░░░░

BARE TABLEAU   Tableau
  clifford bare/cz_many   [0.971-1.001]         0.991x   ██████████████████████████████░┃░░░░
  clifford bare/cnot_many   [0.971-1.020]       1.002x   ██████████████████████████████░┃░░░░

GENERALIZED TABLEAU   GeneralizedTableau
  branch coalesce, 65,536   [0.648-0.676]       0.670x   ████████████████████░░░░░░░░░░░┃░░░░
  clifford gen./cnot_many   [0.941-1.046]       0.994x   ██████████████████████████████░┃░░░░

TABLEAU MIXTURE   GeneralizedTableauMixture
  noisy build   [0.75-0.76]                     0.755x   ███████████████████████░░░░░░░░┃░░░░
  serial sampling, 128 shots   [0.94-0.95]      0.945x   ████████████████████████████░░░┃░░░░
  parallel 8-branch/16-shot   [0.974-1.045]     1.000x   ██████████████████████████████░┃░░░░

SYMBOLIC RING   ppvm-sym-2
  propagation/clifford   [OPEN, 1.05-1.22]      1.100x   ███████████████████████████████╋█░░░

where a process range is given the bar is drawn at its SLOWEST end

This sweep predates the column-major tableau rebuild. Its five tableau rows
describe the row-major -2 frame, not the shipped one; for the tableau engine
the headline table above supersedes them, and it is not uniformly favourable.
The word, Pauli-propagation, mixture and symbolic rows are unaffected — the
rebuild touched ppvm-tableau-2 only.

With that caveat, every end-to-end workload here is at parity or better in
every engine — Trotter, qubit sweeps, branch coalescing, mixture sampling, lossy
GHZ. One row remains genuinely unexplained, a single-gate nanobenchmark family in
the symbolic ring. The Pauli-propagation clifford/cy row that was open at 1.111× is now
fixed at 0.895× (see the fold below), taking its two rows out of the
above-gate bucket. The remaining 64 are adjudicated non-actionable — identical/no-op paths, diagnostic controls,
or layout-sensitive nanobenchmarks — and the largest of those sit in
generalized-tableau observation helpers, where up to 3.7× is +3–6 ns on a path
that is implementation-identical between old and new. Detail in the last two
folds.

Design principles

Compile-time generic throughout. No trait objects, no runtime dispatch for
storage, hashing or policy. Producers are monomorphized type parameters, never
dyn, so the abstraction compiles to nothing on the hot loop.

Decompose by responsibility, not convenience. Config is removed, not
replaced. Each split off Coefficient is justified by a ring that satisfies the
base trait but cannot satisfy the capability — 0.5·(1+i) leaves ℤ[i], which
is why half and Mul<f64> are gone and Halvable exists.

Follow the algebra, don't invent structure. A Pauli mod phase is a vector in
GF(2)^2n; multiplication is XOR up to a ℤ₄ cocycle; conjugation factors into a
symplectic bit map plus a phase. The traits factor the same way, and the symplectic
sign logic is written exactly once:

flowchart TD
  SC["SymplecticColumns<br/>bit-plane algebra · role-independent"]
  PT["PhaseTrack<br/>Z4 for a word · Z2 + g-rule for a tableau"]
  MK["BlanketClifford<br/>opt-in marker"]
  BL["blanket impl Clifford + CliffordExtensions<br/>single audited copy"]
  SC --> BL
  PT --> BL
  MK --> BL
  BL --> W1["PauliWord"]
  BL --> W2["LossyPauliWord<br/>loss-guarded primitives"]
  BL --> W3["Tableau"]
  PH["Phased·W<br/>opts OUT — hand-written fused impl<br/>reads each bit once"]
  SF["StabilizerFrame<br/>role-exclusive · a word never implements it"]
  SF --> W3

Loading

The map is a graded algebra over C[K]. Capabilities layered by algebraic
strength, each admitted by a distinct property and a distinct consumer:
Support (finite partial function) → Accumulate (module core) → Scale
(module action) → Pair (trace pairing) → Multiply (twisted ring product).
Truncation sits deliberately outside: dropping terms still in the support breaks
module exactness, so it lives on Policy via Retain. The key bound is
Indexable, not Word.

A trait must earn its place. Admitted only when generic code consumes it and
there are multiple implementation families, or when it is an established boundary
implemented by different backends. Rejects Config, PauliSumAlgorithm,
TableauMixtureAlgorithm, TableauStorage, and one-alphabet word subtraits.

Word is read-only. Extent, per-site read, weight, iteration. Gate kernels
work at sub-site granularity and are algebra-specific, so mutation lives on each
algebra's own traits — which is what lets a future normal-ordered fermionic
product implement Word honestly.

The finalized digest is first-class. key_hash() is avalanche-quality and
consumed directly as the map hash. The map has no hasher to choose; the key owns
its algorithm, cache and finalization fold, none of which appear in the contract.

No signature leaks array-of-structs. No &mut (K, C), no &mut [C]; mutation
is whole-map or batch. That one rule is what keeps a structure-of-arrays backend
expressible — and ColumnStore is now built on it with zero trait changes.

Randomness is injected, never owned. An owned generator makes a mixture-branch
clone share a stream — a correctness bug that presents as a statistical one — and
stops a value that claims to be pure state from being reproducible from its own
contents. Ownership belongs at the frontend boundary.

Lean is the specification. The machine-checked identities in lean/PPVM/**
are the oracles: phase cocycle, twisted-product associativity, symplectic
isometries, the frame coordinate expansion (Yoder-2012 Lemma 5), the XOR-relabel
bijection, ℓ¹/ℓ² truncation bounds, the Pauli-channel eigenvalue. Where Lean
adjudicates old wrong, the port follows Lean and records it — Projection
computed c²/2 instead of c/2 and left X/Y untouched where completeness
forces dephasing.

Behaviour preservation is otherwise the prime directive. Where old and the
design disagree on anything observable, old wins and the deviation is documented at
the call site. Hence: gates do not truncate, a zero coefficient survives every
operation, and three distinct tableau cutoff spellings are reproduced verbatim
rather than unified.

Additive, with the gate outside the agent. Old and new compile side by side;
differential tests live in a test-only crate depending on both, so shipped -2
crates never depend on old. cargo build, cargo test and lake build are
authoritative pass/fail.

Verification protocol

• Lean-oracle property tests in each -2 crate; differential tests against the old
  crates in ppvm-conformance-2 (~34k lines).
• Ratios are new / old. Improvement below 0.97, parity 0.97–1.03. A regression is
  robust only when the median and the slowest process exceed 1.03 — one
  launch never establishes a regression.
• Harness: mise run perf-report (benchmarks/perf_regression_report.py) drives
  the matrix, pairs medians, and refuses to classify a one-launch row as robust.
• Rows surviving the gate then go through the two -Cllvm-args layout controls in
  benchmarks/README.md, because same-source padding-only rebuilds move several of
  these nanobenchmarks by more than the gate width.
• Downstream: Stim and Vihaco pass all four native feature matrices plus a wasm
  library check. The Python integration is deliberately untouched by this PR —
  ppvm-python-native and ppvm-python are byte-for-byte identical to their
  pre-PR state, the extension still compiles the old crates, and there is no
  backend flag. The traits-2 Python cutover is parked on
  codex/traits-2-python-cutover for a separate PR.

Detail in docs/performance-report.md; per-component gap ledgers in docs/log.md.

Regression benchmark inventory — 17 Criterion targets

All in crates/ppvm-conformance-2/benches; each runs the identical workload
against old and new in one process and reports the ratio. {side} = the paired
old/new arms.

Word layer

target	groups
pauli_word_bench	pauli_word/{product, cnot, key_hash, weight}
lossy_pauli_word_bench	lossy_pauli_word/{product, cnot, key_hash, weight, loss_weight}
phased_pauli_word_bench	phased_pauli_word/{product, cnot}
word_surface_bench	word_surface/{ordinary, lossy, phased, pattern, construct, clone_copy} at 8/64/256 qubits

word_surface covers construction; x_bit/z_bit/get/iter_traverse/weight/
width; set_x_bit/set_z_bit and one-/two-site mutation with clone-then-bits
variants; lossy is_lost/set_lost/clear_loss/set_present/loss_weight/
lossy_present/h_lost_noop; phased add_phase/phase/is_positive/
inner_word/into_parts/from_word_and_phase; cold-cache paths
(new_only_cold_all_components, new_only_cold_first_compute,
new_only_set_x_bit_cold_cache); pattern parse/contains/enumerate_all;
equality, display, hash.

Sum layer

target	groups
pauli_sum_bench	pauli_sum/{build_batch, clifford_h, clifford_x, clifford_cnot, rotation_rx, pauli_error, scale, overlap}
pauli_sum_surface_bench	complete gate/noise surface — see below
pauli_sum_integration	pauli_sum/{rekey_cnot, rx, rzz, pauli_error, truncate, truncate_active} on a growing support
pauli_sum_multiply	pauli_sum/{multiply_word, multiply_sum} (L4 twisted convolution)
pauli_sum_workloads	pauli_sum/{workload_qubit_sweep, workload_random_circuit, workload_trotter_ablation, workload_truncate}
pauli_sum_loss	pauli_sum/loss_interleaved_n12 (neutral-atom GHZ)
pauli_sum_column_store	column_store/{build, clifford_h, clifford_x, clifford_cnot, rx, rx_growth, scale, reduce, overlap, pauli_error, truncate_threshold, truncate_weight, trotter_tfim_n12}
pauli_sum_indexmap	pauli_sum_indexmap/{build, gates, ordered_terms} (Python insertion-order ABI)

pauli_sum_surface_bench:
clifford/{x,y,z,h,s,s_dag,cnot,cz,cy,sqrt_x,sqrt_x_dag,sqrt_y,sqrt_y_dag,cx_alias,zcx_alias,zcy_alias,zcz_alias}
and the clifford_batch/* twins ·
rotation_one/{rx,ry,rz,rot_xy_r} + rotation_one_batch/* ·
rotation_two/rotate_2_generic_xz ·
noise/{pauli_error,two_qubit_pauli_error,depolarize1,depolarize2,amplitude_damping}

• noise_batch/* ·
  pauli_sum_surface/{construct, add, algebra, inspect, truncate, format, new_only} ·
  algebra/scale · projection.

Tableau layer

target	groups
tableau_bench	tableau-micro, tableau-attrib/measure-sweep, tableau-integration/{branch-coalesce, fused-tgate, measure-all-msd, msd-85q, noisy-shots, rot2-brickwork, scaling}
tableau_surface_bench	tableau-surface/{clifford, construction, measurement, noise, observation, projection, rotation, sparse-amplitudes, display}, each split bare/* vs generalized/*
tableau_mixture_bench	mixture/{lifecycle, gate, rotation, measure, reset, noise, scaling, sampler, integration}

tableau_surface_bench covers measure-deterministic/measure-random/
measure_many/measure_all with and without reusable scratch;
compute_decomposition, compute_overlap_case_a, compute_overlap_case_b,
z_expectation, expectation, trace-pattern; cz_block/cz_block_pairs/
cz_block_pairs_cross_word; stabilizer/destabilizer traversal; the amplitude store
(add_or_insert/{hit,miss}, get/{hit,miss}, mul_element_by/{hit,miss},
iteration, clone, equality); measurement record append/overwrite/current;
odd_phase_destabilizer_mask; reset_loss_channel; rotations
one/{rx,ry,rz,rotate_1_x,u3,r_xy} + *_many and two/rotate_2_xz.

tableau_mixture_bench covers lifecycle/{new, new_with_seed, clone, len, is_empty}; scaling/{branches_h, branches_loss, branches_sampler_construction};
sampler/{construction, single_sample, serial_shot_scaling, parallel_branch_shot_scaling, adaptive_sample_shots_128, serial_parallel_crossover}; integration/noisy_build; and
new_only_no_old_twin/* for surface with no old counterpart.

Symbolic layer

target	groups
sym_bench	sym/{micro_term_add, micro_term_mul, micro_mul_term, micro_prod_mul, micro_eval, exact_multiply, expectation_eval, expectation_propagate, random_circuit, random_circuit_clifford, trace_parametric, truncation_sweep, truncation_sweep_eps}
sym_surface	sym/{surface, new_only, api_blocker}

sym_surface covers term construction and arithmetic (add, mul_*, pow,
prod*, promote_*, fold_*_constant, sin_pow, cos_pow, norm_sq),
set_max_sin/set_min_eps, sum accumulation, trace, display, hash, equality,
and old_only/{p0,p1} marking surface the port deliberately does not reproduce.

Plus harness_smoke as the no-op harness sanity target.

Measured results in full

Adjudication of the 75 rows above 1.03 in the 36850446 screening

class	rows
fixed (now below 0.97)	14
parity (0.97–1.03)	11
actionable	0
non-actionable (duplicate path, identical/no-op, or layout control)	50

Plus 5 adjacent grouped rows: 3 fixed, 1 parity, 1 non-actionable, 0 robust.

Where the above-gate non-actionable rows sit, by engine

engine	rows	worst	why non-actionable
generalized tableau	observation/generalized/{bernoulli, flip_with_prob, overwrite_last_measurement_record}	3.71×	identical / no-op path; +2.8–5.9 ns
bare tableau	tableau-micro/{scratch_new_x85, msd_measure_single}	3.65×	diagnostic amortized controls
generalized tableau	noise/generalized/reset_loss_channel	1.22×	representation/layout nanobench; +1.2 ns, stride control 1.048× crossing parity
symbolic ring	sym/surface/construct/{term_variable, term_constant}	3.01×	identical scalar path
word layer	clone_copy/256/lossy/clone_{warm,cold}	2.07×	required three-cache atomic clone
word layer	ordinary/mutate/256/set_{x,z}_bit	1.66×	required immediate cache-valid mutation
word layer	{ordinary,lossy,phased}/read/256/get, observation/equality	1.29×	identical packed read / compare
Pauli propagation	truncate/max_{weight,loss_weight}_disabled, workload_truncate/*/max_sentinel	1.62×	identical disabled path, ~1–2.6 ns
Pauli propagation	inspect/{get, contains_key, contains_key_value}	1.38×	identical primitive
symbolic ring	sym/surface/propagation/clifford/*	1.14×	process-layout crossing (see Still open)

The two tableau/symbolic outliers are the ones worth understanding before trusting
the summary bar: a 3.7× on an operation that is byte-identical between backends is
a placement artifact on a few-nanosecond body, not tableau work getting slower.

Before → after

benchmark	at 36850446	final	process range
pattern contains, ordinary	2.278×	0.015×	0.014–0.016×
pattern contains, lossy-present	1.916×	0.011×	0.011–0.011×
branch coalesce, 65,536	1.863×	0.670×	0.648–0.676×
ordinary CY	1.399×	1.004×	0.998–1.021×
ordinary √X	1.294×	0.870×	0.850–0.893×
pattern parse, indexed	1.151×	0.478×	0.465–0.492×
PauliSum CNOT batch	1.044×	0.853×	0.845–0.919×
PauliSum Z-noise batch	1.037×	0.755×	0.745–0.800×
decomposed-RZZ Trotter	1.094×	0.963×	0.949–0.971×
n=12 sweep	1.070×	1.025×	0.997–1.039×
native RZZ	—	0.960×	median
direct CNOT	—	0.871×	median
phased CNOT / CX / ZCX	— / — / 1.032×	0.980× / 0.985× / 0.989×	medians
mixture parallel 8-branch/16-shot	1.017×	1.000×	0.974–1.045×

Backends

ColumnStore (SoA) vs the hash backend, four fresh processes: steady rx
0.94–0.97×, sustained rx growth 0.93–0.96×, from_terms 0.89–0.92×, active
truncate 0.94–0.95×, realistic rx 0.95–0.98×, native rzz 0.93–0.95×, noisy
TFIM Trotter 0.98–0.99×. New hash vs old on the same Trotter: 0.86–0.90×.

Mixture: noisy build 0.75–0.76×, serial 128-shot sampling 0.94–0.95×.
IndexMapStore: build 0.57×, ordered term export 0.73–0.75×, CNOT 0.81×, rotation
0.89–0.94×. Lossy interleaved n=12 GHZ: 0.99× and 0.84× in two processes.

Still open

• Pauli propagation — pauli_sum_surface/clifford/{cy,zcy_alias}: CLOSED,
  1.107× → 0.895× / 0.889×. The earlier pass had the mechanism right and
  verified the wrong thing — it checked that cy_toggles had no call sites, not
  that the per-term re-key closure had been inlined. It had not; nm still
  showed the symbol with one bl per term. A 32-byte PauliWord exceeds the
  AArch64 register-return limit, so out-of-line meant two plane copies to the
  stack, byte read-modify-writes instead of register toggles, a reload
  overlapping those stores (a forwarding stall on the digest's dependency chain)
  and an indirect sret — ~2 ns/term, which is also why "less memory work, still
  slower" was true. Fix: cy becomes #[inline(never)], so one copy of the
  re-key loop exists, the closure has one call site, and LLVM folds it in. Gate
  body byte-identical.
• Symbolic ring — sym/surface/propagation/*, a 9–10 row family at 1.05–1.22× where the old
  side drifts 10%+ run to run. The fixture is two terms in a map sized from the
  symbolic capacity hint 2^(2n−1) (131,072 buckets), so a single-qubit gate is
  3–6 µs of control-byte sweep both engines must do. Needs a better fixture, not
  more launches.

Why PauliWord hashes eagerly (the design says lazy)

The design's §"Lazy hashing and interior mutability" specifies a lazy,
interior-mutable digest and lists "preserving Copy at the expense of correct
lazy caching" as an explicit non-goal. The shipped ordinary word does the
opposite. That is the outcome of five measured steps, not drift:

commit	change	measured
d2727dd6	Phase 2 ships the design's lazy OnceLock<u64>; Copy dropped as intended	key_hash 6.94 ns cold → 0.60 ns warm
a64ef9f4	OnceLock → relaxed-atomic sentinel	Once's CAS init path dominated the Clifford re-key loop, where every freshly built key hits the cold init exactly once
53ebc66e	warm the branch key's digest one pass earlier	rotation_rx 5.5 → 4.9 µs; ratio 1.07 → ~0.99
b68b7e8e	stop invalidating when a write doesn't change the bit	1.71× → 1.45× and 4.04× → 2.53× on the two affected word benches
3e43610f	ordinary word's digest becomes a plain eager u64, restoring Copy	(see caveat)

The mechanism 53ebc66e established is the load-bearing one, and it was
root-caused rather than guessed: the lazy digest fired inside pass 2's entry(),
on the bucket-index critical path, where the finalize mul-chain stalls the
dependent bucket load. The old crate hashes eagerly in its first pass, so its
probe hits a cached u64 and the hashing overlaps other terms' work. Atomic
barriers (arm64 Relaxed emits none — checked in disassembly), hasher rounds
(IdentityBuildHasher and old's FxBuildHasher do an equal number), resize,
allocation and monomorphization were each ruled out with evidence first.

Clone cost is the other half. word-data-structures.md records the lossy
word's atomic clone at about 2.1× a plain-u64 copy at 256 qubits, and
1f6d7e70 had to add a borrowed-source re-key path (the
PauliBits::PREFER_BORROWED_REKEY seam) purely to dodge that atomic clone. An
eager Copy word does not pay it.

Caveat, stated plainly: 3e43610f is the one step in the chain with no
isolated measurement. It bundles the hash change with cold scratch allocation,
inlined gate paths and direct store operations, so no number attributes the win
to the hash alone. What validates the end state is the 828-pair audit above,
which was run on the shipped eager build.

The eager/lazy split across types is deliberate rather than inconsistent: the
ordinary word is a map key whose digest is read for essentially every instance,
so computing it up front removes a stall and buys back Copy; a Tableau is
invalidated by every Clifford gate and read far less often, so it keeps the
lazy sentinel, and the lossy word additionally needs loss-only mutations not to
rehash the X/Z planes.

Follow-up: the code comments were never updated —
ppvm-pauli-word-2/src/data.rs:44-52 still says "lazy AtomicU64" and "Copy is
intentionally dropped", ppvm-tableau-2/src/data.rs:159 repeats it, and
word-data-structures.md still shows hash_cache: OnceLock<u64>. Those are
wrong and should be corrected; the code is right.

Known deviations from the design docs

Recorded so review can weigh them rather than rediscover them. Each is documented
at its call site with the measurement or behaviour that forced it.

• The graded algebra is unified for one key type, not three. Sum bounds
  S::Key: Word + Indexable, so the tableau mixture and the generalized tableau's
  amplitude vector are specialized types rather than Sum instantiations. The
  amplitude store implements L0/L2/Retain but not Accumulate — a generic
  accumulate_batch over a Vec would be O(m²) on a 65,536-branch state, which is
  exactly what its inline sort-merge exists to avoid.
• The auxiliary map and scratch buffer are back, relocated from Sum into the
  storage backend. The generic producer/batch path clones every key twice; against
  the old in-place aux-map swap that benchmarked ~4.7× slower on a Clifford gate
  and ~2.4× on rx. Clone copies only the primary support.
• Gates do not reduce or truncate. Both are caller-driven — the old crate's
  contract.
• The generic TermBatch key column is a scalar Vec<W>, not W::Column:
  accumulate_batch must be instantiable for the non-Columnar amplitude key. The
  SoA layout is used by ColumnStore's own support. probe_batch probes scalar;
  group prefetching and radix partitioning are not implemented.
• PauliWord's digest cache is eager and the type is Copy again, where the
  design specifies a lazy cache and explicitly trades Copy away for it. This was
  reversed deliberately and under measurement — see the fold below. What is
  genuinely wrong is only the documentation, which still describes the lazy
  form. LossyPauliWord and Tableau remain lazy sentinel-atomic, on purpose.
• PauliStorage and HashFinalize are public, where the design made them
  private to the word crate.
• Cutover is not done. No rename, no deletion; legacy remains the default
  on the Rust frontends, and Python is not wired to -2 at all (see above).
• main is merged in (as of 93d7f7dc), including the breaking
  fix(stim-parser)!: follow Stim's tag model (fix(stim-parser)!: follow Stim's tag model #186). No textual conflicts, and
  no semantic ones either: the whole workspace builds and 1,922 Rust tests plus
  223 Python tests pass. The merge itself touched only stim-parser and
  ppvm-python-native, neither of which is under a conformance benchmark, so it
  moves no number here.
• The tableau's dense-frame measurement is a regression, and it ships. The
  column-major layout buys 4.9× on gates and 15.9× on the surface-code memory
  circuit, and costs 2.7× on the 85-qubit MSD measurement sweep — against both
  legacy and the row-major -2 frame the first audit measured. It is a
  property of the layout, not a bug. A caller that measures one qubit at a time
  rather than in a batch pays 7.8× instead, because it re-orients the frame per
  measurement; holding the orientation lazily would close that, and is not in
  this PR. Weigh it against the workload you care about; the fold above says
  which side each one lands on.
• Benchmark provenance. The headline workloads were measured at 01392466
  and the by-engine sweep at e3a37026; the clifford/cy row alone was
  re-measured at 979efe99, after the fix. That fix does touch a benchmarked
  crate (ppvm-pauli-sum-2), so the rest of the Pauli-propagation numbers were
  taken before it — they are conservative rather than stale: the post-fix
  end-to-end run moved the same direction (rx 0.814×, rzz 0.801×,
  rekey_cnot 0.927×) and no sibling regressed. All tableau rows were
  re-measured at 90902bc6, after the column-major rebuild and the
  measurement-sweep guard fix — the two tables above come from one three-launch
  run of tableau_bench, so they are internally consistent. Every other number
  in this PR predates the rebuild and is unaffected by it; surface_d30,
  cultivation_d5 and the four small STIM programs were measured at 43d6a816,
  which the guard fix does not touch (ppvm-stim already batched M).

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.