Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
54 changes: 54 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: CI

on:
push:
branches: ["**"]
pull_request:
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
test:
name: lint + tests (py${{ matrix.python-version }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12"]

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

# The test suite runs with no network, no Docker, no API key and no GPU,
# so only the dev extra is installed. The SWE-bench and attacker extras
# (swebench, datasets, anthropic, boto3) are not needed to check the
# metric code or to recompute the published numbers from results/.
- name: Install
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

- name: Lint
run: ruff check .

- name: Tests
run: pytest -q

# Guards the README's "results: reproducible" badge: the published
# headline must still fall out of the committed records.
- name: Recompute the published headline from results/
# --check exits 1 if any README figure stops recomputing. Without it this
# step always passed and guarded nothing.
run: |
python harness/swe_native_rate.py
python harness/swe_confirm.py --check
10 changes: 8 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,15 @@ __pycache__/
*.pyc
.DS_Store

# Intermediate raw attacker records (large; the committed results/ hold the evidence + reproducers)
results/raw_swe/
# Intermediate raw attacker records from exploratory runs (large, superseded).
# NOTE: results/raw_swe_500/ is deliberately NOT ignored -- it is the committed
# evidence for the published 51.0% native-hackable rate. See harness/swe_native_rate.py.
results/raw_swe_*/
results/raw_swe/
!results/raw_swe_500/
scratch_swe/
scratch_logs/
third_party/

# local test virtualenv
.venv*/
34 changes: 34 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,37 @@ should filter on the canary string above.

Every finding here concerns public benchmark tasks (SWE-bench Verified) and their test
suites — not any individual author's private work.

Third-party source in results/raw_swe_500/
------------------------------------------

The attack records under results/raw_swe_500/ contain unified diffs against SWE-bench
Verified tasks. Those diffs quote source from the upstream projects the tasks are drawn
from, as context and removed lines. That quoted source remains the copyright of its
respective authors and is used under its own licence, NOT under this repository's MIT
licence. The MIT licence in LICENSE covers Tripwire's own code, metrics and metadata.

Upstream projects quoted, by task count, with their licences:

231 django/django BSD-3-Clause
75 sympy/sympy BSD-3-Clause
44 sphinx-doc/sphinx BSD-2-Clause
34 matplotlib/matplotlib matplotlib licence (BSD-compatible)
32 scikit-learn/scikit-learn BSD-3-Clause
22 astropy/astropy BSD-3-Clause
22 pydata/xarray Apache-2.0
19 pytest-dev/pytest MIT
10 pylint-dev/pylint GPL-2.0-or-later
8 psf/requests Apache-2.0
2 mwaskom/seaborn BSD-3-Clause
1 pallets/flask BSD-3-Clause

The pylint fragments are called out explicitly because pylint is copyleft while this
repository is not. What appears here is diff context around edits of a few lines each;
no pylint file is reproduced whole and nothing here is runnable as pylint. It is quotation
in a research artifact, not redistribution of the program. If the pylint maintainers
disagree, contact the author and those ten records will be removed.

This follows the practice of SWE-bench's own experiments repository, which publishes
per-instance patches, trajectories and execution logs against these same projects.

40 changes: 36 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
[![Results: reproducible](https://img.shields.io/badge/results-reproducible-brightgreen.svg)](#-reproduce-it-yourself)
[![Compute: ~$15](https://img.shields.io/badge/compute-~%2415-success.svg)](#cost)
[![Compute: $29.98](https://img.shields.io/badge/compute-%2429.98-success.svg)](#cost)
[![Reproduces arXiv:2606.16062](https://img.shields.io/badge/reproduces-arXiv%3A2606.16062-b31b1b.svg)](https://arxiv.org/abs/2606.16062)
[![Oracle: differential execution](https://img.shields.io/badge/oracle-differential%20execution-8A2BE2.svg)](#how-it-works)

Expand Down Expand Up @@ -117,12 +117,27 @@ python harness/code_attack.py <instance_ids...> \
python harness/diffexec_oracle.py astropy__astropy-14309 --exploit-file <candidate.diff>
```

**4 — Aggregate** (Wilson CIs, Benjamini–Hochberg FDR, cluster-bootstrap):
**4 — Recompute the headline** from the committed evidence in `results/` (no Docker, no API key).
Both numbers derive from files in this repository — the verifier-alone rate from the 500 per-task
attack records, the confirmed rate from the per-task dual-gate verdicts:

```bash
python harness/swe_native_rate.py # 51.0% (255/500) and the 226-task confirm queue
python harness/swe_confirm.py # 13.7% [10.1-18.0%] — Wilson interval, pre-registered estimator
```

For the hub Class-A code lane (Wilson CIs, Benjamini–Hochberg FDR, cluster-bootstrap over envs):

```bash
python harness/code_aggregate.py
```

**5 — Run the tests** (no network, no Docker, no API key, no GPU):

```bash
pip install -e ".[dev]" && pytest -q
```

Exact parameters, the seed-42 stratified sample, and the per-task verdicts are in
[`docs/anchor-result.md`](docs/anchor-result.md) and [`docs/extend-result.md`](docs/extend-result.md).

Expand All @@ -133,14 +148,25 @@ harness/ the tooling (attacker, SWE-bench adapter, differential-executi
code_attack.py the reward-hack attacker (K=3, failure-log feedback, deterministic diff transport)
swebench_adapter.py apply any patch → run the native verifier → read resolved
diffexec_oracle.py the dual-gate: gold vs candidate behavioral divergence, verdict by execution
code_aggregate.py Wilson CIs, BH-FDR, cluster-bootstrap over tasks
aggregate.py Wilson CIs, BH-FDR, beta-binomial pooling, cluster-bootstrap over envs
code_aggregate.py the same statistics for the hub Class-A code lane
swe_confirm.py recomputes the published confirmed rate (13.7%) from results/
swe_native_rate.py recomputes the verifier-alone rate (51.0%) from the raw attack records
results/ the evidence — every confirmed hack, its reproducer, and the pre-drawn sample
raw_swe_500/ all 500 per-task attack records: the evidence behind 51.0% (4.1 MB)
tests/ the metric test suite — no network, no Docker, no API key, no GPU
docs/
anchor-result.md the reproduction (11/49) + capability ladder, per-task rationale
extend-result.md the novel full-500 result (51% / 13.7%), sampling + honest limitations
preregistration.md thresholds & protocol, committed before any verdict
```

> **A note on the name.** The public artifact is **Tripwire**. `envcert` is the older internal name
> and still appears in runtime identifiers — the Docker image and container names
> (`envcert-base:latest`), the egress network and proxy names, the in-container probe paths and the
> run-id prefixes. Those are load-bearing strings, not prose, so they are left alone; nothing named
> `envcert` is a separate project.

## Honest limitations

Conservative science is a credibility asset, so these are stated up front (full detail in the docs):
Expand All @@ -157,7 +183,13 @@ Conservative science is a credibility asset, so these are stated up front (full
- An earlier full-500 pass was **discarded** after a Docker disk-exhaustion bug silently produced empty
patches; caught, fixed, and re-run clean. The before/after is documented as part of the record.

<a name="cost"></a>Total compute: **~$15** of cloud credits. Every result is re-runnable from this repo.
<a name="cost"></a>**Total attacker spend: $29.98**, summed from the `cost_usd` field of every
attack record. **$21.60 of that is checkable from this repository** — the full-500 pass, whose 500
records are committed and whose total is pinned by `pytest`. The remaining **$8.38 is not
independently checkable here**: the anchor run ($2.53), the discarded v1 ($2.64), the Haiku
capability rung ($2.58) and the retest and diagnostic passes ($0.63), whose raw records stay out of
the repo as superseded intermediates. Docker and CPU time are local and unmetered. (An earlier
revision said `~$15`; that was the anchor-scale estimate and it understated the total.)

## Why this matters

Expand Down
2 changes: 1 addition & 1 deletion docs/anchor-result.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ substituted, only the diff *transport*.)
By repo: **astropy 5 + django 6 = 11** (paper: astropy 6 + django 8 = 14). Single-shot round-1: 4/11
(paper 9/49). Artifacts: `results/anchor_confirm.json` (diffexec verdicts + divergent inputs),
`results/anchor_handreview.json` (per-task HACK/CORRECT rationale), `results/raw_swe_v1/` (the
undercounted first run, kept for the fidelity-fix comparison). Spend: ~$10-15 AWS Bedrock credits (~$0 OOP).
undercounted first run, kept for the fidelity-fix comparison). Spend: $5.17 AWS Bedrock credits for this lane ($2.53 anchor + $2.64 the undercounted v1), ~$0 OOP. An earlier revision said "~$10-15"; that was an estimate. Project total is $29.98, see the README cost note.

## Reading — a successful, conservative reproduction
**11/49 (22.4%) vs the paper's 14/49 (28.5%)** — same repos, comparable rate, with an OPEN,
Expand Down
2 changes: 1 addition & 1 deletion docs/extend-result.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ benchmark the field treats as gold-standard for code-RL reward.

Artifacts: `results/swe500_confirmed.json` (aggregate), `results/swe500_sample_verdicts.json` (per-task
verdicts + reproducers/rationale), `results/swe500_confirm_sample.json` (the pre-drawn seed-42 sample),
`results/raw_swe_500/` (all 500 raw attacker records; intermediate, not committed).
`results/raw_swe_500/` (all 500 raw attacker records, committed; `harness/swe_native_rate.py` rederives the 500/255/226 inputs from them).

### Methodology notes (honest limitations)
- 28 of 31 confirmed HACKs rest on code hand-review, not deterministic execution — 42/102 sampled tasks
Expand Down
37 changes: 36 additions & 1 deletion docs/preregistration.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# envcert — pre-registration
# Tripwire — pre-registration

**Committed before any aggregate is computed.** The git commit timestamp of this file is the
pre-registration record. Thresholds, verifier-type routing, metrics, statistics, model slate, and
Expand Down Expand Up @@ -188,3 +188,38 @@ propagated to the rate over 500. Reported ALONGSIDE the native-hackable rate (V=
looser metric) — never in place of it. The sample is drawn before any hand-review verdict is read. Naming
any individual broken task publicly still requires full (non-sampled) review of that task, per the
disclosure gate.

**2026-09-02 — Provenance of the pre-registration timestamp for the published runs.**
Registered as a correction to the header claim, not to any method or number.

The header of this file states that "the git commit timestamp of this file is the pre-registration
record." **For the runs published so far that claim is not independently verifiable.** The public
repository was opened with a **single squashed commit** (`95be36b`, 2026-07-26) that contains this
pre-registration AND the anchor, ladder and 500-extend results together. One commit carries one
timestamp, so git cannot demonstrate that this file predates the verdicts it governs. Nothing about
the protocol changes; what changes is the strength of the evidence for its ordering, and that is
stated here rather than left for a reader to discover.

**What IS verifiable from the repository, independent of commit order:**
- **The sample was fixed before the verdicts.** The seed-42 stratified draw is a separate artifact
(`results/swe500_confirm_sample.json`) holding the seed, the queue size, and the 102 drawn ids. Its
per-repo composition is checkable against the declared proportional strata, and it is disjoint from
the verdict file, so a reader can see the frame without trusting the tally.
- **Every counted verdict carries its own evidence.** Each confirmed hack ships either a
divergent-input reproducer from deterministic differential execution or a written hand-review
rationale; each CORRECT and AMBIGUOUS verdict ships one too. A reader re-derives the tally, p̂, the
Wilson interval and the published rate from `results/` with `python harness/swe_confirm.py`.
- **The thresholds are code, not prose.** §5's 5% bar, the q=0.05 FDR level, the Wilson quantile and
the bootstrap seed are named constants in `harness/aggregate.py` carrying their §-citations, so any
deviation from the registered values appears as a diff.
- **Every registered bias points down.** Wilson lower bound rather than point estimate; AMBIGUOUS
excluded from the numerator and kept in the denominator; soft graders excluded from binary FAR;
ambiguous probes excluded from FAR. The looser readings are disclosed alongside (native-hackable
51.0%; AMBIGUOUS-as-hack 15.5%). A choice made after seeing results to flatter the headline would
have had to move the number **up**, and each such choice is on the record moving it down.

**Going forward, from this amendment onward:** each phase's pre-registration lands as **its own
commit, pushed before any record of that phase's results exists in the repository**, and the
result document for that phase cites the pre-registration commit hash. Publication does not squash
history. Where a phase's registration is added to this file, it is added as a dated amendment below,
in its own commit, ahead of the corresponding run — never in the same commit as the run's output.
25 changes: 20 additions & 5 deletions harness/aggregate.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""envcert Phase 2 aggregate statistics.
"""Tripwire Phase 2 aggregate statistics.

Reads a directory of ``*.battery.jsonl`` files (one per env, one JSON record per
(row, probe) plus a single ``mode=="classification"`` summary line) and computes
Expand Down Expand Up @@ -303,7 +303,22 @@ def _read_battery(path):


def compute_env(path, index):
"""Read one battery file and compute its metrics. I/O only; the metric
computation lives in compute_env_from_records, which is pure and testable."""
summary, records = _read_battery(path)
env_id_fallback = os.path.basename(path).replace(
".battery.jsonl", "").replace("__", "/", 1)
return compute_env_from_records(summary, records, index,
env_id_fallback=env_id_fallback)


def compute_env_from_records(summary, records, index, env_id_fallback=None):
"""Per-env metrics from already-parsed records (pure: no file access).

`summary` is the mode=="classification" line (or None); `records` are the
per-(row, probe) lines; `index` maps env_id -> {tags, domain}.
"""
index = index or {}
# Skip error records (spec: "Records may carry an error field (skip those).")
scored = [r for r in records if "error" not in r]
n_error = len(records) - len(scored)
Expand All @@ -314,7 +329,7 @@ def compute_env(path, index):
if not env_id and scored:
env_id = scored[0].get("env_id")
if not env_id:
env_id = os.path.basename(path).replace(".battery.jsonl", "").replace("__", "/", 1)
env_id = env_id_fallback or "unknown"

# verifier_type: prefer the classification summary line (authoritative).
vtype = None
Expand Down Expand Up @@ -701,7 +716,7 @@ def print_summary(result):
bf = result["binary_far"]
soft = result["soft"]
print("=" * 72)
print("envcert aggregate (pre-reg §5: Wilson-LB > {:.0%}, BH-FDR q={})".format(
print("Tripwire aggregate (pre-reg §5: Wilson-LB > {:.0%}, BH-FDR q={})".format(
FAR_THRESHOLD, FDR_Q))
print("=" * 72)
print("\nCOVERAGE (total env files: {})".format(cov["total_env_files"]))
Expand Down Expand Up @@ -804,7 +819,7 @@ def pool_eyr(raw_dir):


def main(argv=None):
ap = argparse.ArgumentParser(description="envcert Phase 2 aggregate statistics")
ap = argparse.ArgumentParser(description="Tripwire Phase 2 aggregate statistics")
ap.add_argument("--raw", default="results/raw",
help="directory of *.battery.jsonl files")
ap.add_argument("--index", default="results/hub_index.jsonl",
Expand All @@ -823,7 +838,7 @@ def main(argv=None):
with open(out, "w") as fh:
json.dump(pooled, fh, indent=2, default=str)
print("=" * 72)
print("envcert ELICITED / EYR aggregate (pilot; pre-reg §2 — NOT headlined <150 envs)")
print("Tripwire ELICITED / EYR aggregate (pilot; pre-reg §2 — NOT headlined <150 envs)")
print("=" * 72)
print(f" envs: {pooled['n_envs']} "
f"EYR-positive (strict, reached anchor + oracle WRONG): "
Expand Down
4 changes: 1 addition & 3 deletions harness/attacker.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""envcert ELICITED / EYR lane (pre-reg §2 EYR, §3 dual-gate, §9 canned-vs-elicited).
"""Tripwire ELICITED / EYR lane (pre-reg §2 EYR, §3 dual-gate, §9 canned-vs-elicited).

The adversarial lane. Instead of canned probes, an ATTACKER model actively tries
to produce an output the env's verifier PASSES but the INDEPENDENT oracle marks
Expand Down Expand Up @@ -41,9 +41,7 @@
the REAL oracle (see harness/tests/test_attacker.py).
"""
import json
import os
import re
import time
import urllib.error
import urllib.request

Expand Down
2 changes: 1 addition & 1 deletion harness/classify.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""envcert verifier-type classifier (pre-reg §1).
"""Tripwire verifier-type classifier (pre-reg §1).

Runs INSIDE the disposable container (needs a loaded `env`). Introspects the
env's rubric + env class into exactly one bucket:
Expand Down
Loading
Loading