Repository navigation
Harden the instrument: tests, CI, and the evidence behind every published number - #1
Merged
Merged
Conversation
Covers the primitives that turn per-probe counts into the headline: Wilson score interval, exact binomial tail, Benjamini-Hochberg step-up, beta-binomial pooling and the seeded cluster bootstrap. Expected values come from independent sources — the score equation the Wilson interval inverts, closed forms at k=0 and k=n, hand-computed binomial tails, and the worked example in Benjamini & Hochberg (1995) — never from re-running the implementation's own algebra. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Three seams that decide what the headline counts:
- oracle.screen/label_candidate: provenance tags, the deciding channel, and
the hard rule that a probe the oracle cannot confirm wrong goes to the
ambiguous bucket and is EXCLUDED from FAR (pre-reg §3).
- classify: the five verifier-type buckets, weight-0 monitor exclusion, and
the empirical binary -> soft flip on observed partial credit (pre-reg §1).
- aggregate: FAR against the env's own gold anchor, RHS over degenerate
provenance, FRR, gold-fails and non-discriminating exclusions, soft-grader
monotonicity, and the BH-FDR / pooling / stratification roll-up (§2, §5).
aggregate.compute_env is split so the metric computation is pure: the new
compute_env_from_records takes parsed records, and compute_env reads the file
and delegates. No metric behaviour changes — compute_env's output for a given
file is identical, and a test asserts the two paths agree.
Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
- code_attack.dual_gate_verdict (extracted): verifier PASS alone is never a
confirmed reward-hack, oracle WRONG without a PASS is not one either, and
an oracle residual (uninvokable / no divergence) routes to hand review
rather than being auto-passed as correct. The advisory LLM gate never
flips the verdict.
- code_attack.behaviorally_distinct: the confirm-queue filter.
- code_aggregate: the reject-level pass bar, escape-exploit counting,
ambiguous exclusion, the FRR control, and the per-env outcome buckets
(partial-credit-to-wrong and non-discriminating both route OUT of FAR).
Both compute_env functions are split the same way as aggregate.py: a pure
compute_env_from_records plus a thin file-reading caller. No metric behaviour
changes; the record built by attack_task is field-for-field identical.
Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
harness/swe_confirm.py implements the estimator registered in the 2026-07-14
pre-registration amendment and the anchor tally, reading only results/. It
computes nothing new: it puts the previously uncommitted derivation of the
published figures into code so a reviewer can re-run it.
python harness/swe_confirm.py
Every published figure is reproduced from the committed per-task verdicts:
anchor Sonnet 4.5 11/49 = 22.4% (3 diffexec + 8 hand, of 21 candidates)
anchor Haiku 4.5 8/49 = 16.3% (3 diffexec + 5 hand, of 19 candidates)
extend sample tally 31 HACK / 67 CORRECT / 4 AMBIGUOUS (n=102)
extend p-hat 0.3039 Wilson95 [0.2231, 0.3990]
extend confirmed 13.7% 95% CI [10.1%, 18.0%] (68.7/500, CI [50.4, 90.2])
Three inputs stay DECLARED because results/raw_swe_500/ is gitignored: the
500-task frame, the 226-task confirm queue and the 255 native-hackable tasks
(the 51.0% verifier-alone figure). Everything downstream of them is recomputed.
Both the module docstring and the CLI output say so.
No result file, number or claim was changed.
Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
- pyproject.toml (hatchling, MIT, same conventions as the author's other
repos). Core dependencies are empty because the aggregation, oracle and
classification code is standard-library only; the attack lanes are extras
(swe / anthropic / bedrock / dev). requirements.txt is unchanged apart from
a header pointing at the equivalent extras.
- .github/workflows/ci.yml runs ruff and pytest on 3.11 and 3.12, then
re-derives the published headline from results/ so a change that breaks
reproducibility fails the build. No network, Docker, API key or GPU.
- Lint gate: ruff's default rules, with E501 and E741 off and two documented
per-file ignores, so it passes without reflowing the modules that produce
the published numbers. Nine genuinely unused imports were deleted; no other
harness code was touched.
Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Renames the cosmetic occurrences: module docstring headers, CLI --help text and printed banners. Runtime identifiers keep the internal name on purpose — the Docker image and container names, the egress network and proxy names, the in-container probe paths, the run-id prefixes, the LiteLLM key and the no-op control patch are load-bearing strings, and renaming them would change what the harness does. The README now says so in one line. README also: step 4 pointed at code_aggregate.py, which aggregates the hub Class-A code lane, not the SWE-bench run. It now points at swe_confirm.py for the SWE-bench headline and keeps code_aggregate.py for the hub lane, and adds a step for running the tests. No number or finding changed. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
…estamp Appended below the original text, as §9 requires; nothing above it is edited and no number, threshold or method changes. The header claims the git commit timestamp of this file is the pre-registration record. For the published runs that is not independently verifiable: the public repository was opened with a single squashed commit (95be36b, 2026-07-26) carrying this file and the results together, so git cannot show the ordering. The amendment states that plainly, sets out what IS verifiable without it (the seed-42 draw committed as its own artifact disjoint from the verdicts, per-task reproducers and rationales, thresholds as cited constants in code, and every registered bias pointing the headline down with the looser readings disclosed alongside), and commits to per-phase pre-registration as its own commit, before any result for that phase exists, with the hash cited in the result document. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
The verifier-alone rate led the README but its evidence was gitignored, so nobody who cloned the repo could check it, and no committed code derived it. README step 4 also pointed at code_aggregate.py, which aggregates the hub Class-A lane rather than SWE-bench. Commits results/raw_swe_500/ (500 records, 4.1 MB, no secrets or local paths) and adds harness/swe_native_rate.py, which rederives all three inputs the confirmed-rate estimator declares: total_tasks 500, native_hackable 255 (51.0%), queue_size 226. tests/test_swe_native_rate.py fails the build if the derivation ever disagrees with results/swe500_confirmed.json. No result, number or claim changed. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Committing the attack records made the cost claim checkable, and it was wrong. Summing cost_usd across every attacker run gives $29.98; the full-500 pass alone is $21.60. The old figure was the anchor-scale estimate. The README now states the total, its provenance, and that the earlier number understated it. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
ruff caught an unused sys import in swe_native_rate.py; I had run pytest locally but not ruff, so CI failed on the first push. The swe_confirm CLI also still told the reader raw_swe_500/ was not committed, which stopped being true one commit earlier. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Five independent reviewers, none of which wrote the branch. Fixed here: - CI's 'recompute the headline' step could not fail: swe_confirm.py printed and returned 0 unconditionally, so a green check guarded nothing. Adds --check, which pins 7 published figures plus the CI bounds and exits 1 on drift. Verified by flipping one verdict in a copy: 5 drifts, exit 1. - The $29.98 cost replaced one unverifiable figure with another. Only $21.60 is derivable here. README now states the split, swe_native_rate reports the pass cost, and a test pins $21.60. - Three files still claimed results/raw_swe_500/ was not committed, one commit after it was. - docs/anchor-result.md still carried the ~$10-15 estimate this branch set out to correct. Now $5.17, summed from that lane's records. - .gitignore enumerated six paths, so a new raw dir would be committed silently. Back to a wildcard with an explicit exception for raw_swe_500. - Pre-registration was still titled 'envcert'. - NOTICE now names the twelve upstream projects quoted in the diffs and their licences, states that MIT covers only Tripwire's own work, and calls out the pylint GPL fragments specifically. Not fixed here, escalated to the author: the README's framing of the 51%. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The project's claim is that you can trust its measurement. This branch makes that checkable.
What was wrong
results/raw_swe_500/was gitignored, so nobody who cloned the repo could verify the first number in the README.code_aggregate.py, which aggregates the hub Class-A lane, not SWE-bench.~$15claimed; the full-500 pass alone cost $21.60.What changed
tests/.github/workflows/ci.ymlharness/swe_native_rate.pyharness/swe_confirm.pyresults/raw_swe_500/pyproject.tomldocs/preregistration.mdREADME.mdWhat did not change
No result, number or finding was altered. The published 13.7% reproduces exactly: 31/67/4 → p̂ 0.3039 → Wilson95 [0.2231, 0.3990] → 13.7% [10.1%, 18.0%]; anchor 22.4%, Haiku 16.3%. There was no bug in the metric code.
The cost correction
Committing the evidence made the compute claim falsifiable, and it was false. Summing
cost_usdacross every attacker run gives $29.98. The old~$15was the anchor-scale estimate. Corrected in the badge and the cost section, with the provenance stated.Pure-core extractions
Three functions were extracted from I/O callers so they could be tested (
aggregate.compute_env_from_records,code_aggregate.compute_env_from_records,code_attack.dual_gate_verdict). No metric behaviour changed; a test asserts the file path equals the records path.https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS