diff --git a/CHANGELOG.md b/CHANGELOG.md index b2b0e89..be520a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ Follows the first real-agent A/B (bench/agent-ab, pull request #77), which found - `query` adds one line naming the next ranked files and the `explain` and `path` commands when the question words suggest a flow or chain, or the top files are scattered with flat scores. It stays inside the token budget and leaves single-answer queries unchanged. `STACKLIT_QUERY_NEXT` (0 turns it off), `STACKLIT_FLOW_WORDS`, `STACKLIT_QUERY_SPREAD_DIRS`, `STACKLIT_QUERY_SPREAD_TOP`, `STACKLIT_QUERY_FLATNESS`. - `impact` on a file or directory lists its importers as `path:line`, with the sources checked for script languages so the list matches a grep, and `--calls` for the previous call-level view. A Go package directory used to be an error. About 40 to 160 tokens for a package against 110 to 390 for `grep -rn` of the same import. `STACKLIT_IMPACT_SCAN`, `STACKLIT_IMPACT_SCAN_MAX`. - Small repo mode: below `small_repo_files` (default 600, `STACKLIT_SMALL_REPO_FILES`, 0 off) source files, the session hook, subagent hook and `setup` carry a one line note that grep is enough instead of the digest (about 60 tokens instead of 490). -- Quiet repo mode: below `quiet_repo_files` (default 5000, `STACKLIT_QUIET_REPO_FILES`, 0 off) source files, the session and subagent hooks inject nothing at all. Round 2 measured Stacklit as costlier at 412 to 2,919 files and cheaper only at 15,978, so the 5000 default sits between and is not yet measured. The `setup` rules block stays the one line note. +- Quiet repo mode: below `quiet_repo_files` (default 5000, `STACKLIT_QUIET_REPO_FILES`, 0 off) source files, the session and subagent hooks inject nothing at all. Round 2 measured Stacklit as costlier at 412 to 2,919 files and cheaper only at 15,978, so the 5000 default sits between and is not yet measured. The `setup` rules block stays the one line note. Round 3 (`bench/agent-ab/RESULTS-round3.md`) measured it: below 5,000 files a quiet session still cost 1.08 times a session without Stacklit [1.05, 1.11], against 1.05 to 1.42 before, so it does not remove the cost. - `impact` prints a `trust:` line saying how many listed dependents are sure (score 0.9 or more) and how many need a confirming grep. - `path` adds advice after the weakest hop score: sure (0.9 or more) needs no confirming grep, otherwise grep the weak hop. - A pre-registered second A/B round in `bench/agent-ab/tasks-round2/`, with a `stacklit-e` arm in the harness. It has not been run. diff --git a/bench/agent-ab/RESULTS-round3.md b/bench/agent-ab/RESULTS-round3.md new file mode 100644 index 0000000..d43aae2 --- /dev/null +++ b/bench/agent-ab/RESULTS-round3.md @@ -0,0 +1,40 @@ +# Round 3 results: does quiet repo mode remove the cost below 5,000 files? + +Pre-registration: `PREREGISTRATION-round3.md`. Raw records: `results-round3/`; ledger, summary and tables: `round3/`. Model claude-sonnet-5-5, 300 sessions (control 150, stacklit-quiet 150), 3 repos, 10 tasks each (the round 2 tasks), 5 trials. Notional spend 5.95 USD of the 15 USD cap. No harness failures, rate limits or stopped runs. Bootstrap seed 20260508, 10,000 resamples, paired by task and trial. + +## Verdict + +Quiet mode does not pass. Q1 fails and Q3 fails on theauth. Q2 passes. No claim of "no regression below 5,000 files" is supported, and the default quiet limit is not shown to be right. + +| Id | Criterion | Result | Outcome | +|---|---|---|---| +| Q1 | Pooled cost ratio, upper bound below 1.05 | 1.08 [1.05, 1.11] | fail | +| Q2 | Correctness difference, lower bound above -5 points | 100% in both arms (150/150) | pass | +| Q3 | Per repo cost ratio, upper bound below 1.15 | theauth-go 1.04 [1.00, 1.09], featuredrop 1.07 [1.02, 1.11], theauth 1.13 [1.09, 1.18] | fail (theauth, 1.18) | + +## Other metrics, quiet against control (ratio, pooled, 150 pairs) + +| Metric | control | stacklit-quiet | ratio [95% CI] | +|---|---|---|---| +| cost (USD) | 0.0191 | 0.0206 | 1.08 [1.05, 1.11] | +| total tokens | 16,925 | 21,710 | 1.28 [1.23, 1.34] | +| uncached-equivalent tokens | 5,252 | 5,974 | 1.14 [1.11, 1.17] | +| turns | 3.19 | 3.09 | 0.97 [0.92, 1.01] | +| tool calls | 2.19 | 2.09 | 0.95 [0.89, 1.02] | + +The agent ran zero stacklit commands in 150 of 150 treatment sessions, and the session hook injected nothing. So the extra tokens do not come from stacklit output or from the hook text. They come from what the arm still carries: the one line rules block that `setup` writes, and the Bash tool the arm is given (control has Read, Grep and Glob only). This round cannot tell which of the two causes the cost. That is a hypothesis, not a finding. + +## Compared with round 2 (stacklit-e against control, different sessions) + +theauth-go 1.05 to 1.04, featuredrop 1.21 to 1.07, theauth 1.42 to 1.13. Quiet mode removed most of the extra cost but not all of it. These are separate runs, so the comparison is indicative only. + +## Deviations and limits + +- The analyzer did not know the arm name `stacklit-quiet`, so its first pass wrote empty tables. `treatmentArms` in `analyze.go` was extended and the analysis re-run on the same ledger. No data changed. +- The tasks are the round 2 tasks, run again. featuredrop was contaminated in round 2's memorisation check (3 of 6 right with tools off). Without featuredrop the pooled ratio is not computed here. +- Correctness is at ceiling (100% in both arms), so Q2 cannot detect a small loss. +- One model, one machine, tasks written by us on repos we own, notional cost. + +## What this means for the default + +Leave quiet mode in, since it cut the cost, but do not describe it as free. The next step is a diagnostic arm that separates the two causes: control plus the Bash tool with no stacklit setup, and quiet with no rules block. It needs its own pre-registration and approval. diff --git a/bench/agent-ab/analyze.go b/bench/agent-ab/analyze.go index 081aebf..80beaff 100644 --- a/bench/agent-ab/analyze.go +++ b/bench/agent-ab/analyze.go @@ -151,7 +151,7 @@ func loadTasks(dir, repoName string) map[string]task { } // treatmentArms lists every arm that is compared with its base. -var treatmentArms = []string{"stacklit", "strict", "stacklit-e", "stacklit-e-small"} +var treatmentArms = []string{"stacklit", "strict", "stacklit-e", "stacklit-e-small", "stacklit-quiet"} func pairs(c config, repoName, arm string) []pair { return pairsAgainst(c, repoName, arm, "control") } diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t1.json new file mode 100644 index 0000000..e4b7bce --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 4, + "cost_usd": 0.0320306, + "usage": { + "input_tokens": 6, + "output_tokens": 467, + "cache_creation_input_tokens": 6397, + "cache_read_input_tokens": 17606 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.137060583, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS|\u003crss|atom", + "Grep pattern=manifest", + "Grep pattern=rss path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:08Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t2.json new file mode 100644 index 0000000..1c407de --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 3, + "cost_usd": 0.0255315, + "usage": { + "input_tokens": 4, + "output_tokens": 273, + "cache_creation_input_tokens": 5478, + "cache_read_input_tokens": 8815 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.09440925, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS", + "Grep pattern=manifest" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:56Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t3.json new file mode 100644 index 0000000..ee2ad38 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts, src/bridges.ts", + "turns": 4, + "cost_usd": 0.030179599999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 504, + "cache_creation_input_tokens": 5844, + "cache_read_input_tokens": 17516 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.545461625, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS", + "Grep pattern=manifest", + "Grep pattern=rss path=src/bridges.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t4.json new file mode 100644 index 0000000..b2ac0fd --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 4, + "cost_usd": 0.020183399999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 437, + "cache_creation_input_tokens": 3572, + "cache_read_input_tokens": 15134 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.810286583, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|\u003cfeed|atom", + "Grep pattern=rss path=src/index.ts", + "Grep pattern=rss path=src/rss.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:34Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t5.json new file mode 100644 index 0000000..eb6ceb3 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-01-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 5, + "cost_usd": 0.0232428, + "usage": { + "input_tokens": 8, + "output_tokens": 629, + "cache_creation_input_tokens": 3708, + "cache_read_input_tokens": 21048 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.970601667, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS", + "Grep pattern=manifest path=/private/tmp/abreal/work/ab-1691852961/co-2542119967/src", + "Grep pattern=rss|manifest path=/private/tmp/abreal/work/ab-1691852961/co-2542119967/src/rss.ts", + "Grep pattern=rss path=/private/tmp/abreal/work/ab-1691852961/co-2542119967/src/bridges.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:43Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t1.json new file mode 100644 index 0000000..f87d5e6 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.0215878, + "usage": { + "input_tokens": 8, + "output_tokens": 493, + "cache_creation_input_tokens": 3625, + "cache_read_input_tokens": 21418 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.731618458, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=topological|toposort|cycle|circular", + "Grep pattern=topological|toposort|cycle path=src", + "Grep pattern=topological|toposort|cycle path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:45Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t2.json new file mode 100644 index 0000000..1f3dc6e --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.0215878, + "usage": { + "input_tokens": 8, + "output_tokens": 493, + "cache_creation_input_tokens": 3625, + "cache_read_input_tokens": 21418 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.452275167, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=topological|toposort|cycle|circular", + "Grep pattern=topological|toposort|cycle path=src", + "Grep pattern=topological|toposort|cycle path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:59Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t3.json new file mode 100644 index 0000000..8a40d7d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.021378099999999997, + "usage": { + "input_tokens": 8, + "output_tokens": 477, + "cache_creation_input_tokens": 3613, + "cache_read_input_tokens": 21401 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.968828167, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=topological|toposort|cycle|circular", + "Grep pattern=topological|toposort|cycle path=src", + "Grep pattern=topological|toposort|cycle path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t4.json new file mode 100644 index 0000000..a878b47 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.0212969, + "usage": { + "input_tokens": 8, + "output_tokens": 433, + "cache_creation_input_tokens": 3704, + "cache_read_input_tokens": 21349 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.994787333, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)topological|cycle|circular", + "Grep pattern=(?i)topological|cycle|circular path=src", + "Grep pattern=(?i)topological|cycle|circular path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:28Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t5.json new file mode 100644 index 0000000..5072585 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-02-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.021678400000000004, + "usage": { + "input_tokens": 8, + "output_tokens": 477, + "cache_creation_input_tokens": 3686, + "cache_read_input_tokens": 21484 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.303246834, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=topological|toposort|topoSort|circular|cycle", + "Grep pattern=topolog|cycle path=src", + "Grep pattern=topolog|cycle path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:28Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t1.json new file mode 100644 index 0000000..3d16c1b --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 3, + "cost_usd": 0.0196685, + "usage": { + "input_tokens": 6, + "output_tokens": 360, + "cache_creation_input_tokens": 3646, + "cache_read_input_tokens": 14725 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.923831042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=/private/tmp/abreal/work/ab-1691852961/co-317564344" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:21Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t2.json new file mode 100644 index 0000000..4a982ba --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.0205629, + "usage": { + "input_tokens": 6, + "output_tokens": 429, + "cache_creation_input_tokens": 3697, + "cache_read_input_tokens": 14729 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.7254885, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=src/size-budget.ts", + "Grep pattern=gzip path=scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:38Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t3.json new file mode 100644 index 0000000..685f9fa --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.020224899999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 404, + "cache_creation_input_tokens": 3675, + "cache_read_input_tokens": 14729 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.594518416, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=src/size-budget.ts", + "Grep pattern=gzip path=scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t4.json new file mode 100644 index 0000000..8bd7557 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.0194889, + "usage": { + "input_tokens": 6, + "output_tokens": 360, + "cache_creation_input_tokens": 3601, + "cache_read_input_tokens": 14729 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.931370459, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=src/size-budget.ts", + "Grep pattern=gzip path=scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:49Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t5.json new file mode 100644 index 0000000..57d3c2e --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-03-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.0202609, + "usage": { + "input_tokens": 6, + "output_tokens": 408, + "cache_creation_input_tokens": 3674, + "cache_read_input_tokens": 14729 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.1916225, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip|brotli", + "Grep pattern=gzip path=src/size-budget.ts", + "Grep pattern=gzip path=scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:12Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t1.json new file mode 100644 index 0000000..ab9942a --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 3, + "cost_usd": 0.0179853, + "usage": { + "input_tokens": 4, + "output_tokens": 235, + "cache_creation_input_tokens": 3686, + "cache_read_input_tokens": 8833 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.959587125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=**/*{nboarding,hecklist}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t2.json new file mode 100644 index 0000000..4bf3379 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 4, + "cost_usd": 0.0191511, + "usage": { + "input_tokens": 6, + "output_tokens": 299, + "cache_creation_input_tokens": 3652, + "cache_read_input_tokens": 15411 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.140498042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=src/**/*{hecklist,egistry}*", + "Glob pattern=src/**/*{se}Checklist*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:45Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t3.json new file mode 100644 index 0000000..82ae277 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t3.json @@ -0,0 +1,41 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 5, + "cost_usd": 0.0224964, + "usage": { + "input_tokens": 6, + "output_tokens": 348, + "cache_creation_input_tokens": 4363, + "cache_read_input_tokens": 15524 + }, + "tool_calls": { + "Glob": 3, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.217126167, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=**/*[Oo]nboarding*", + "Glob pattern=src/**/*[Cc]hecklist*", + "Glob pattern=src/react/**/*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:19Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t4.json new file mode 100644 index 0000000..d76c5d3 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 3, + "cost_usd": 0.0179853, + "usage": { + "input_tokens": 4, + "output_tokens": 235, + "cache_creation_input_tokens": 3686, + "cache_read_input_tokens": 8833 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.251463667, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=**/*{nboarding,hecklist}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:32Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t5.json new file mode 100644 index 0000000..e50369d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-04-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 4, + "cost_usd": 0.021506699999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 295, + "cache_creation_input_tokens": 4251, + "cache_read_input_tokens": 15407 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.003664292, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=src/**/*{hecklist,hecklist-}*", + "Glob pattern=src/react/**/*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t1.json new file mode 100644 index 0000000..28645c0 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0138707, + "usage": { + "input_tokens": 4, + "output_tokens": 190, + "cache_creation_input_tokens": 2770, + "cache_read_input_tokens": 8827 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.379671583, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t2.json new file mode 100644 index 0000000..5ccc347 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0138707, + "usage": { + "input_tokens": 4, + "output_tokens": 190, + "cache_creation_input_tokens": 2770, + "cache_read_input_tokens": 8827 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.578850041, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:34Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t3.json new file mode 100644 index 0000000..cf6a0a3 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0138867, + "usage": { + "input_tokens": 4, + "output_tokens": 192, + "cache_creation_input_tokens": 2769, + "cache_read_input_tokens": 8827 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.184004625, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t4.json new file mode 100644 index 0000000..569668f --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0138707, + "usage": { + "input_tokens": 4, + "output_tokens": 190, + "cache_creation_input_tokens": 2770, + "cache_read_input_tokens": 8827 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.855649042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:20Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t5.json new file mode 100644 index 0000000..041fe2a --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-05-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0138707, + "usage": { + "input_tokens": 4, + "output_tokens": 190, + "cache_creation_input_tokens": 2770, + "cache_read_input_tokens": 8827 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.798440875, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:21Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t1.json new file mode 100644 index 0000000..a93ce8b --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.0121416, + "usage": { + "input_tokens": 4, + "output_tokens": 105, + "cache_creation_input_tokens": 2550, + "cache_read_input_tokens": 8836 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.300334416, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**/*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:26Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t2.json new file mode 100644 index 0000000..aff4cd4 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.012585600000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 149, + "cache_creation_input_tokens": 2551, + "cache_read_input_tokens": 8836 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.848862917, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:08Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t3.json new file mode 100644 index 0000000..f4b5e65 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.012523399999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 144, + "cache_creation_input_tokens": 2548, + "cache_read_input_tokens": 8834 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.22399925, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**/*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:41Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t4.json new file mode 100644 index 0000000..376be6d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.0121256, + "usage": { + "input_tokens": 4, + "output_tokens": 103, + "cache_creation_input_tokens": 2551, + "cache_read_input_tokens": 8836 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.457354875, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t5.json new file mode 100644 index 0000000..1d904d8 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-06-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.0127116, + "usage": { + "input_tokens": 4, + "output_tokens": 162, + "cache_creation_input_tokens": 2550, + "cache_read_input_tokens": 8836 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.988330833, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**/*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:30Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t1.json new file mode 100644 index 0000000..de889fb --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/banner.tsx, src/react/components/changelog-widget.tsx, src/react/components/announcement-modal.tsx, src/rss.ts, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.0179279, + "usage": { + "input_tokens": 4, + "output_tokens": 417, + "cache_creation_input_tokens": 3217, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.606535625, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown(\\.js|\\.ts)", + "Glob pattern=**/markdown.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:07Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t2.json new file mode 100644 index 0000000..d8c72c5 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/rss.ts, src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/changelog-widget.tsx, src/react/components/banner.tsx, src/react/components/announcement-modal.tsx, src/react/components/changelog-page.tsx, src/react/components/spotlight.tsx, src/vue/components/changelog-widget.ts, src/vue/components/toast.ts, src/vue/components/spotlight.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.0175479, + "usage": { + "input_tokens": 4, + "output_tokens": 389, + "cache_creation_input_tokens": 3192, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.216248541, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/markdown.ts", + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown|import\\(['" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:03Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t3.json new file mode 100644 index 0000000..3c2c8c0 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/components/announcement-modal.tsx, src/react/components/banner.tsx, src/rss.ts, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.0174919, + "usage": { + "input_tokens": 4, + "output_tokens": 385, + "cache_creation_input_tokens": 3188, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.931331917, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown|import\\(['", + "Glob pattern=**/markdown.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:35Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t4.json new file mode 100644 index 0000000..797c20b --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/components/announcement-modal.tsx, src/react/components/banner.tsx, src/rss.ts, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.0179597, + "usage": { + "input_tokens": 4, + "output_tokens": 419, + "cache_creation_input_tokens": 3220, + "cache_read_input_tokens": 8817 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.494043167, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown(\\.js|\\.ts)", + "Glob pattern=**/src/markdown.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:29Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t5.json new file mode 100644 index 0000000..db38d07 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-07-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/rss.ts, src/react/components/banner.tsx, src/react/components/changelog-widget.tsx, src/react/components/announcement-modal.tsx, src/react/components/changelog-page.tsx, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.0174919, + "usage": { + "input_tokens": 4, + "output_tokens": 385, + "cache_creation_input_tokens": 3188, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.958279583, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown|import\\(['", + "Glob pattern=**/markdown.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:50Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t1.json new file mode 100644 index 0000000..b6f34b3 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 3, + "cost_usd": 0.015425900000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 296, + "cache_creation_input_tokens": 2894, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.384123375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]\\)|", + "Glob pattern=**/i18n.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:17Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t2.json new file mode 100644 index 0000000..19dd5b5 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/context.ts, src/react/components/changelog-widget.tsx", + "turns": 3, + "cost_usd": 0.0164479, + "usage": { + "input_tokens": 4, + "output_tokens": 391, + "cache_creation_input_tokens": 2912, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.556348834, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/i18n*", + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]\\)|" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:31Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t3.json new file mode 100644 index 0000000..b26a96c --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/context.ts, src/react/components/changelog-widget.tsx, src/react/components/changelog-page.tsx", + "turns": 3, + "cost_usd": 0.015395900000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 293, + "cache_creation_input_tokens": 2894, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.293724292, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.[jt]s)?['\"]|require\\(['\"][^'\"]*i18n(\\.[jt]s)?['\"]\\)|impo", + "Glob pattern=**/i18n.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:28Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t4.json new file mode 100644 index 0000000..758771e --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 3, + "cost_usd": 0.0155199, + "usage": { + "input_tokens": 4, + "output_tokens": 293, + "cache_creation_input_tokens": 2925, + "cache_read_input_tokens": 8819 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.229353333, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=i18n", + "Glob pattern=**/i18n*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:15Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t5.json new file mode 100644 index 0000000..464a0ca --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-08-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 3, + "cost_usd": 0.0154397, + "usage": { + "input_tokens": 4, + "output_tokens": 293, + "cache_creation_input_tokens": 2905, + "cache_read_input_tokens": 8817 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.680750584, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/i18n*", + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.[jt]s)?['\"]|require\\(['\"][^'\"]*i18n(\\.[jt]s)?['\"]\\)|impo" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:21Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t1.json new file mode 100644 index 0000000..daec860 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/toast.tsx, src/react/index.ts", + "turns": 3, + "cost_usd": 0.017890200000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 264, + "cache_creation_input_tokens": 3428, + "cache_read_input_tokens": 15262 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.435929833, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=[Tt]oast", + "Grep pattern=[Tt]oast path=src/react/index.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:04Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t2.json new file mode 100644 index 0000000..802e162 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/toast.tsx, src/react/index.ts", + "turns": 3, + "cost_usd": 0.0186761, + "usage": { + "input_tokens": 6, + "output_tokens": 292, + "cache_creation_input_tokens": 3552, + "cache_read_input_tokens": 15361 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.18247975, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=[Tt]oast", + "Grep pattern=[Tt]oast path=/private/tmp/abreal/work/ab-1691852961/co-1122693654/src/react/index.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:34Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t3.json new file mode 100644 index 0000000..d68dc5d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/\u003cnew-component\u003e.tsx, src/react/index.ts", + "turns": 3, + "cost_usd": 0.01833, + "usage": { + "input_tokens": 6, + "output_tokens": 269, + "cache_creation_input_tokens": 3523, + "cache_read_input_tokens": 15360 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.327077334, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=[Tt]oast", + "Grep pattern=[Tt]oast path=src/react/index.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:20Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t4.json new file mode 100644 index 0000000..59fe04c --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/toast.tsx, src/react/index.ts", + "turns": 4, + "cost_usd": 0.019726, + "usage": { + "input_tokens": 6, + "output_tokens": 347, + "cache_creation_input_tokens": 3674, + "cache_read_input_tokens": 15480 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.680440417, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=[Tt]oast", + "Glob pattern=packages/*/src/index.*", + "Grep pattern=[Tt]oast path=/private/tmp/abreal/work/ab-1691852961/co-1878046772/src/react/index.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:21Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t5.json new file mode 100644 index 0000000..5b2c764 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-09-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/\u003cname\u003e.tsx, src/react/index.ts", + "turns": 4, + "cost_usd": 0.0180664, + "usage": { + "input_tokens": 6, + "output_tokens": 417, + "cache_creation_input_tokens": 3100, + "cache_read_input_tokens": 14844 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.600946959, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*oast* path=/private/tmp/abreal/work/ab-1691852961/co-3731141480", + "Glob pattern=packages/*/src/index.ts* path=/private/tmp/abreal/work/ab-1691852961/co-3731141480", + "Grep pattern=toast path=/private/tmp/abreal/work/ab-1691852961/co-3731141480/src/react" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t1.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t1.json new file mode 100644 index 0000000..6bcb1ac --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.0169581, + "usage": { + "input_tokens": 6, + "output_tokens": 243, + "cache_creation_input_tokens": 3263, + "cache_read_input_tokens": 14641 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.984500417, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:47Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t2.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t2.json new file mode 100644 index 0000000..86bba0f --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.0169581, + "usage": { + "input_tokens": 6, + "output_tokens": 243, + "cache_creation_input_tokens": 3263, + "cache_read_input_tokens": 14641 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.165029708, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:39Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t3.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t3.json new file mode 100644 index 0000000..cf690c7 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.0182574, + "usage": { + "input_tokens": 6, + "output_tokens": 243, + "cache_creation_input_tokens": 3588, + "cache_read_input_tokens": 14634 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.442708917, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:38Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t4.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t4.json new file mode 100644 index 0000000..5e5b836 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.0186821, + "usage": { + "input_tokens": 6, + "output_tokens": 271, + "cache_creation_input_tokens": 3624, + "cache_read_input_tokens": 14641 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.749128958, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=/private/tmp/abreal/work/ab-1691852961/co-2140625278/src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:57Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t5.json b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t5.json new file mode 100644 index 0000000..1bcaee0 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/control/fd2-10-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.0169581, + "usage": { + "input_tokens": 6, + "output_tokens": 243, + "cache_creation_input_tokens": 3263, + "cache_read_input_tokens": 14641 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.910239708, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:52Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t1.json new file mode 100644 index 0000000..7d8ad94 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 3, + "cost_usd": 0.029403199999999997, + "usage": { + "input_tokens": 4, + "output_tokens": 288, + "cache_creation_input_tokens": 6337, + "cache_read_input_tokens": 11672 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.262857125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS|\u003cfeed|atom", + "Grep pattern=manifest" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t2.json new file mode 100644 index 0000000..b2cbffc --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts, src/bridges.ts", + "turns": 4, + "cost_usd": 0.021228200000000003, + "usage": { + "input_tokens": 6, + "output_tokens": 449, + "cache_creation_input_tokens": 3700, + "cache_read_input_tokens": 19262 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.172831375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS", + "Grep pattern=manifest path=/private/tmp/abreal/work/ab-1691852961/co-956722871/src/rss.ts", + "Grep pattern=rss path=/private/tmp/abreal/work/ab-1691852961/co-956722871/src/bridges.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t3.json new file mode 100644 index 0000000..e341d94 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 3, + "cost_usd": 0.019877600000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 263, + "cache_creation_input_tokens": 4018, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.647755875, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS", + "Grep pattern=manifest" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:56Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t4.json new file mode 100644 index 0000000..e855be1 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 3, + "cost_usd": 0.019877600000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 263, + "cache_creation_input_tokens": 4018, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.563563375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|RSS", + "Grep pattern=manifest" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:50Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t5.json new file mode 100644 index 0000000..d959719 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-01-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-01", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/rss.ts", + "turns": 3, + "cost_usd": 0.018760500000000003, + "usage": { + "input_tokens": 6, + "output_tokens": 319, + "cache_creation_input_tokens": 3410, + "cache_read_input_tokens": 19185 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.509931417, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=rss|\u003cfeed|\u003cchannel", + "Grep pattern=manifest path=/private/tmp/abreal/work/ab-1691852961/co-2336682392/src/rss.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:52Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t1.json new file mode 100644 index 0000000..e608eae --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.0233545, + "usage": { + "input_tokens": 8, + "output_tokens": 473, + "cache_creation_input_tokens": 3975, + "cache_read_input_tokens": 27085 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.060520375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cycle|topolog|circular", + "Grep pattern=cycle|topolog|circular path=src", + "Grep pattern=cycle|topolog|circular path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:10Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t2.json new file mode 100644 index 0000000..5b1792a --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 3, + "cost_usd": 0.020276, + "usage": { + "input_tokens": 6, + "output_tokens": 333, + "cache_creation_input_tokens": 3752, + "cache_read_input_tokens": 19260 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.571775792, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cycle|circular|topolog", + "Grep pattern=cycle|circular|topolog path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t3.json new file mode 100644 index 0000000..9732a4d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.023392, + "usage": { + "input_tokens": 8, + "output_tokens": 473, + "cache_creation_input_tokens": 3984, + "cache_read_input_tokens": 27100 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.623876667, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cycle|topolog|circular", + "Grep pattern=cycle|topolog|circular path=src", + "Grep pattern=cycle|topolog|circular path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:06Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t4.json new file mode 100644 index 0000000..3d152f9 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.0233545, + "usage": { + "input_tokens": 8, + "output_tokens": 473, + "cache_creation_input_tokens": 3975, + "cache_read_input_tokens": 27085 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.819733333, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cycle|circular|topolog", + "Grep pattern=cycle|circular|topolog path=src", + "Grep pattern=cycle|circular|topolog path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:19Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t5.json new file mode 100644 index 0000000..b3ca773 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-02-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-02", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/dependencies.ts", + "turns": 4, + "cost_usd": 0.0233581, + "usage": { + "input_tokens": 8, + "output_tokens": 498, + "cache_creation_input_tokens": 3913, + "cache_read_input_tokens": 27101 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.729128375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cycle|topological|toposort", + "Grep pattern=cycle|topological|toposort path=src", + "Grep pattern=cycle|topological|toposort path=src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t1.json new file mode 100644 index 0000000..1c674fd --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.0227353, + "usage": { + "input_tokens": 6, + "output_tokens": 410, + "cache_creation_input_tokens": 4174, + "cache_read_input_tokens": 19273 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.680330333, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=src/size-budget.ts", + "Grep pattern=gzip path=scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t2.json new file mode 100644 index 0000000..532c248 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 3, + "cost_usd": 0.022494, + "usage": { + "input_tokens": 6, + "output_tokens": 399, + "cache_creation_input_tokens": 4142, + "cache_read_input_tokens": 19240 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.7043535, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=/private/tmp/abreal/work/ab-1691852961/co-3877145947" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t3.json new file mode 100644 index 0000000..8cfd202 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.023411500000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 456, + "cache_creation_input_tokens": 4228, + "cache_read_input_tokens": 19275 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.637012791, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=/private/tmp/abreal/work/ab-1691852961/co-3371184081/src/size-budget.ts", + "Grep pattern=gzip path=/private/tmp/abreal/work/ab-1691852961/co-3371184081/scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:35Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t4.json new file mode 100644 index 0000000..b17edb5 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.022916699999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 421, + "cache_creation_input_tokens": 4192, + "cache_read_input_tokens": 19267 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.036411041, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=src/size-budget.ts", + "Grep pattern=gzip path=scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:10Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t5.json new file mode 100644 index 0000000..4a7be94 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-03-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-03", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "src/size-budget.ts, scripts/check-bundle-budgets.mjs", + "turns": 4, + "cost_usd": 0.0226275, + "usage": { + "input_tokens": 6, + "output_tokens": 400, + "cache_creation_input_tokens": 4172, + "cache_read_input_tokens": 19275 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.81242325, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=gzip", + "Grep pattern=gzip path=src/size-budget.ts", + "Grep pattern=gzip path=scripts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:27Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t1.json new file mode 100644 index 0000000..86aaf1f --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t1.json @@ -0,0 +1,40 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 4, + "cost_usd": 0.0221282, + "usage": { + "input_tokens": 6, + "output_tokens": 296, + "cache_creation_input_tokens": 4302, + "cache_read_input_tokens": 19482 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.7538227079999995, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=src/react/**/*", + "Glob pattern=src/**/*{hecklist,egistry}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:10Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t2.json new file mode 100644 index 0000000..bf54a92 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 4, + "cost_usd": 0.0230354, + "usage": { + "input_tokens": 6, + "output_tokens": 364, + "cache_creation_input_tokens": 4347, + "cache_read_input_tokens": 19954 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.646989792, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=src/**/*{hecklist,hecklist-registry}*", + "Grep pattern=registry path=src/react" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t3.json new file mode 100644 index 0000000..d19cd1d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 3, + "cost_usd": 0.018660200000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 244, + "cache_creation_input_tokens": 3565, + "cache_read_input_tokens": 19482 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.217416583, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=src/**/*{hecklist,egistry}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t4.json new file mode 100644 index 0000000..e064868 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 3, + "cost_usd": 0.018660200000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 244, + "cache_creation_input_tokens": 3565, + "cache_read_input_tokens": 19482 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.323324958, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=src/**/*{hecklist,egistry}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t5.json new file mode 100644 index 0000000..bff35f6 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-04-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-04", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/checklist.tsx, src/react/hooks/use-checklist.ts, src/react/checklist-registry.ts", + "turns": 3, + "cost_usd": 0.018660200000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 244, + "cache_creation_input_tokens": 3565, + "cache_read_input_tokens": 19482 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.700748959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=onboarding", + "Glob pattern=src/**/*{hecklist,egistry}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:18Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t1.json new file mode 100644 index 0000000..3cbed0d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0159926, + "usage": { + "input_tokens": 4, + "output_tokens": 184, + "cache_creation_input_tokens": 3244, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.87746825, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t2.json new file mode 100644 index 0000000..b97512a --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0159926, + "usage": { + "input_tokens": 4, + "output_tokens": 184, + "cache_creation_input_tokens": 3244, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.593972708, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:43Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t3.json new file mode 100644 index 0000000..0b7eac8 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0159926, + "usage": { + "input_tokens": 4, + "output_tokens": 184, + "cache_creation_input_tokens": 3244, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.401510792, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:04Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t4.json new file mode 100644 index 0000000..fd4ae5b --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0159926, + "usage": { + "input_tokens": 4, + "output_tokens": 184, + "cache_creation_input_tokens": 3244, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.31758225, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t5.json new file mode 100644 index 0000000..01f72de --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-05-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-05", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/feature-request-button.tsx, src/react/components/feature-request-form.tsx, src/react/feature-request-store.ts", + "turns": 2, + "cost_usd": 0.0159762, + "usage": { + "input_tokens": 4, + "output_tokens": 184, + "cache_creation_input_tokens": 3240, + "cache_read_input_tokens": 11682 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.954563959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=feature.?request" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:47Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t1.json new file mode 100644 index 0000000..5e845f3 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.014791499999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 147, + "cache_creation_input_tokens": 3036, + "cache_read_input_tokens": 11695 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.232408208, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**/*.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:05Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t2.json new file mode 100644 index 0000000..58a5bc8 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.014801499999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 148, + "cache_creation_input_tokens": 3036, + "cache_read_input_tokens": 11695 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.5000522499999995, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**/*.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:08Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t3.json new file mode 100644 index 0000000..471ad24 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.014751499999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 143, + "cache_creation_input_tokens": 3036, + "cache_read_input_tokens": 11695 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.516155333, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**/*.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:21:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t4.json new file mode 100644 index 0000000..3e99d2e --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 2, + "cost_usd": 0.0147815, + "usage": { + "input_tokens": 4, + "output_tokens": 146, + "cache_creation_input_tokens": 3036, + "cache_read_input_tokens": 11695 + }, + "tool_calls": { + "Glob": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.142448791, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/engine/**/*.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:40Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t5.json new file mode 100644 index 0000000..172e40c --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-06-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-06", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/engine/adoption-scorer.ts, src/engine/behavior-tracker.ts, src/engine/timing-optimizer.ts", + "turns": 3, + "cost_usd": 0.0165909, + "usage": { + "input_tokens": 6, + "output_tokens": 219, + "cache_creation_input_tokens": 3120, + "cache_read_input_tokens": 19089 + }, + "tool_calls": { + "Bash": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.56449425, + "seven_day_utilization": 0.69, + "call_log": [ + "Bash find . -type d -name engine -not -path \"*/node_modules/*\" | head; ", + "Bash ls -R src/engine" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:41Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t1.json new file mode 100644 index 0000000..ae95625 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/components/announcement-modal.tsx, src/react/components/banner.tsx, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts, src/rss.ts", + "turns": 3, + "cost_usd": 0.0198354, + "usage": { + "input_tokens": 4, + "output_tokens": 396, + "cache_creation_input_tokens": 3675, + "cache_read_input_tokens": 11674 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.2850913330000004, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown(\\.js)?['\"]", + "Glob pattern=**/markdown.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:24:43Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t2.json new file mode 100644 index 0000000..6663dbd --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/rss.ts, src/react/components/announcement-modal.tsx, src/react/components/banner.tsx, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.019741799999999997, + "usage": { + "input_tokens": 4, + "output_tokens": 387, + "cache_creation_input_tokens": 3674, + "cache_read_input_tokens": 11678 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.18490025, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/markdown.ts", + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown|import\\(['" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:44Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t3.json new file mode 100644 index 0000000..20d9043 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/announcement-modal.tsx, src/rss.ts, src/react/components/banner.tsx, src/react/components/changelog-widget.tsx, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.0195158, + "usage": { + "input_tokens": 4, + "output_tokens": 372, + "cache_creation_input_tokens": 3655, + "cache_read_input_tokens": 11678 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.285881625, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/markdown.ts", + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:58Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t4.json new file mode 100644 index 0000000..ea2d139 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/components/announcement-modal.tsx, src/react/components/banner.tsx, src/rss.ts, src/vue/components/banner.ts, src/vue/components/toast.ts, src/vue/components/spotlight.ts, src/vue/components/changelog-widget.ts", + "turns": 2, + "cost_usd": 0.0187198, + "usage": { + "input_tokens": 4, + "output_tokens": 332, + "cache_creation_input_tokens": 3556, + "cache_read_input_tokens": 11678 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.789525042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown|import\\(['" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:09Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t5.json new file mode 100644 index 0000000..99b7c19 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-07-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-07", + "kind": "impact", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/smart-announcement.tsx, src/react/components/toast.tsx, src/react/components/spotlight.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/components/announcement-modal.tsx, src/react/components/banner.tsx, src/rss.ts, src/vue/components/changelog-widget.ts, src/vue/components/spotlight.ts, src/vue/components/toast.ts, src/vue/components/banner.ts", + "turns": 3, + "cost_usd": 0.0203238, + "usage": { + "input_tokens": 4, + "output_tokens": 446, + "cache_creation_input_tokens": 3672, + "cache_read_input_tokens": 11678 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.631422625, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/markdown.ts", + "Grep pattern=from\\s+['\"][^'\"]*markdown(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*markdown|import\\(['" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:24Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t1.json new file mode 100644 index 0000000..db6bbca --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 3, + "cost_usd": 0.0210284, + "usage": { + "input_tokens": 6, + "output_tokens": 460, + "cache_creation_input_tokens": 3623, + "cache_read_input_tokens": 19244 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.539029125, + "seven_day_utilization": 0.69, + "call_log": [ + "Bash git ls-files | grep -E 'i18n' ; grep -rnE \"from ['\\\"][^'\\\"]*i18n(\\.ts|\\.js)?['\\\"]|require\\(['\\\"][^'\\\"]*i18n|import\\(['\\\"][^'\\\"]*i18n\" --include=*.ts --include=*", + "Grep pattern=i18n path=/private/tmp/abreal/work/ab-1691852961/co-3106307300" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:24Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t2.json new file mode 100644 index 0000000..7cff256 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 3, + "cost_usd": 0.017001799999999997, + "usage": { + "input_tokens": 4, + "output_tokens": 249, + "cache_creation_input_tokens": 3334, + "cache_read_input_tokens": 11678 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.985778958, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*i18n", + "Glob pattern=**/i18n.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:45Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t3.json new file mode 100644 index 0000000..c27d60b --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 3, + "cost_usd": 0.0178514, + "usage": { + "input_tokens": 4, + "output_tokens": 326, + "cache_creation_input_tokens": 3354, + "cache_read_input_tokens": 11674 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.1190295, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.[jt]s)?['\"]|require\\(['\"][^'\"]*i18n|import\\(['\"][^'\"]*i1", + "Glob pattern=**/i18n*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:19:34Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t4.json new file mode 100644 index 0000000..3b6a6fc --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 3, + "cost_usd": 0.0169694, + "usage": { + "input_tokens": 4, + "output_tokens": 249, + "cache_creation_input_tokens": 3326, + "cache_read_input_tokens": 11674 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.579006, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*i18n", + "Glob pattern=**/i18n.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:40Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t5.json new file mode 100644 index 0000000..3232271 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-08-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-08", + "kind": "impact", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/provider.tsx, src/react/components/changelog-page.tsx, src/react/components/changelog-widget.tsx, src/react/context.ts", + "turns": 2, + "cost_usd": 0.0175458, + "usage": { + "input_tokens": 4, + "output_tokens": 331, + "cache_creation_input_tokens": 3265, + "cache_read_input_tokens": 11678 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.365291375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=from\\s+['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]|require\\(['\"][^'\"]*i18n(\\.js|\\.ts)?['\"]\\)|" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:20:52Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t1.json new file mode 100644 index 0000000..6599f3d --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/\u003cnew-component\u003e.tsx, src/react/index.ts", + "turns": 3, + "cost_usd": 0.0212633, + "usage": { + "input_tokens": 6, + "output_tokens": 337, + "cache_creation_input_tokens": 3975, + "cache_read_input_tokens": 19813 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.3857995, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=toast", + "Grep pattern=toast path=/private/tmp/abreal/work/ab-1691852961/co-3198638637/src/react/index.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:07Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t2.json new file mode 100644 index 0000000..9cd9a62 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/\u003cname\u003e.tsx, src/react/index.ts", + "turns": 3, + "cost_usd": 0.0208878, + "usage": { + "input_tokens": 6, + "output_tokens": 310, + "cache_creation_input_tokens": 3949, + "cache_read_input_tokens": 19798 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.249155584, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=toast", + "Grep pattern=[Tt]oast path=/private/tmp/abreal/work/ab-1691852961/co-409211650/src/react/index.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t3.json new file mode 100644 index 0000000..bc7087e --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/new-announcement-style.tsx, src/react/index.ts", + "turns": 2, + "cost_usd": 0.017219699999999998, + "usage": { + "input_tokens": 4, + "output_tokens": 153, + "cache_creation_input_tokens": 3628, + "cache_read_input_tokens": 11697 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.021634958, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=toast" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:17:20Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t4.json new file mode 100644 index 0000000..9f7ed79 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/toast.tsx, src/react/index.ts", + "turns": 2, + "cost_usd": 0.0175173, + "usage": { + "input_tokens": 4, + "output_tokens": 174, + "cache_creation_input_tokens": 3650, + "cache_read_input_tokens": 11693 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.7476312499999995, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=toast" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t5.json new file mode 100644 index 0000000..7c98091 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-09-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-09", + "kind": "add", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/react/components/toast.tsx, src/react/index.ts\n\nThose are the paths for the existing toast. For a new component, the file would go in `src/react/components/`, and `src/react/index.ts` would re-export it.", + "turns": 3, + "cost_usd": 0.0215449, + "usage": { + "input_tokens": 6, + "output_tokens": 375, + "cache_creation_input_tokens": 3951, + "cache_read_input_tokens": 19789 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.70225225, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=toast", + "Grep pattern=toast path=/private/tmp/abreal/work/ab-1691852961/co-2871924778/src/react/index.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:18:35Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t1.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t1.json new file mode 100644 index 0000000..a7d6e40 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.020633899999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 243, + "cache_creation_input_tokens": 4069, + "cache_read_input_tokens": 19159 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.178295292, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t2.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t2.json new file mode 100644 index 0000000..f20382c --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.0206507, + "usage": { + "input_tokens": 6, + "output_tokens": 243, + "cache_creation_input_tokens": 4073, + "cache_read_input_tokens": 19167 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.090042375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:22:12Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t3.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t3.json new file mode 100644 index 0000000..ec1025e --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.019304699999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 242, + "cache_creation_input_tokens": 3739, + "cache_read_input_tokens": 19167 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.194088875, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:16:31Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t4.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t4.json new file mode 100644 index 0000000..cf76128 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.0206367, + "usage": { + "input_tokens": 6, + "output_tokens": 242, + "cache_creation_input_tokens": 4072, + "cache_read_input_tokens": 19167 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.442948375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:23:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t5.json b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t5.json new file mode 100644 index 0000000..3f96e28 --- /dev/null +++ b/bench/agent-ab/results-round3/featuredrop/stacklit-quiet/fd2-10-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "featuredrop", + "task": "fd2-10", + "kind": "add", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "src/cli.ts", + "turns": 3, + "cost_usd": 0.019304699999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 242, + "cache_creation_input_tokens": 3739, + "cache_read_input_tokens": 19167 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.495193334, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=generate-rss", + "Grep pattern=generate-rss path=src/cli.ts" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:40Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t1.json new file mode 100644 index 0000000..af435f3 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t1.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 5, + "cost_usd": 0.029527400000000002, + "usage": { + "input_tokens": 8, + "output_tokens": 654, + "cache_creation_input_tokens": 5187, + "cache_read_input_tokens": 22234 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.518107667, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(limiter|ratelimit|rate_limit)", + "Grep pattern=(?i)(evict|idle|lastSeen|last_used|sweep) path=/private/tmp/abreal/work/ab-2259468618/co-2537392846/internal/ratelimit", + "Grep pattern=(?i)(evict|idle|lastSeen|sweep) path=/private/tmp/abreal/work/ab-2259468618/co-2537392846/internal/throttle", + "Grep pattern=(?i)(evict|idle|lastSeen|sweep|^type |^func New) path=/private/tmp/abreal/work/ab-2259468618/co-2537392846/internal" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:03Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t2.json new file mode 100644 index 0000000..686cd8d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 5, + "cost_usd": 0.0289188, + "usage": { + "input_tokens": 8, + "output_tokens": 728, + "cache_creation_input_tokens": 4853, + "cache_read_input_tokens": 22108 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.792479125, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(limiter|ratelimit|rate_limit)", + "Grep pattern=(?i)evict|idle|lastUsed|lastSeen|sweep|janitor path=/private/tmp/abreal/work/ab-2259468618/co-379965800/internal", + "Grep pattern=(?i)evict|idle|lastUsed|lastSeen|sweep|janitor|per.key|email path=/private/tmp/abreal/work/ab-2259468618/co-379965800/internal/ratelimit/ratelimit", + "Grep pattern=(?i)evict|idle|lastUsed|lastSeen|sweep|janitor path=/private/tmp/abreal/work/ab-2259468618/co-379965800/internal/throttle/memory.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:05:06Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t3.json new file mode 100644 index 0000000..a477a3f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t3.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 5, + "cost_usd": 0.0287332, + "usage": { + "input_tokens": 8, + "output_tokens": 736, + "cache_creation_input_tokens": 4786, + "cache_read_input_tokens": 22132 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.095919875, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(limiter|ratelimit|rate_limit)", + "Grep pattern=(?i)evict|idle|lastSeen|last_used|janitor|sweep path=/private/tmp/abreal/work/ab-2259468618/co-3702588783", + "Grep pattern=(?i)evict|idle|lastSeen|last_used|janitor|sweep|per.key|email|\\bIP\\b path=/private/tmp/abreal/work/ab-2259468618/co-3702588783/internal/ratelimit/ratelimi", + "Grep pattern=(?i)evict|idle|lastSeen|last_used|janitor|sweep|per.key|email|\\bIP\\b path=/private/tmp/abreal/work/ab-2259468618/co-3702588783/internal/throttle/memory.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t4.json new file mode 100644 index 0000000..352e635 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 4, + "cost_usd": 0.028745200000000002, + "usage": { + "input_tokens": 8, + "output_tokens": 555, + "cache_creation_input_tokens": 5241, + "cache_read_input_tokens": 22152 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.183598584, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)limiter", + "Grep pattern=(?i)evict|idle|sweep|gc path=/private/tmp/abreal/work/ab-2259468618/co-4170755525/internal", + "Grep pattern=(?i)evict|idle|sweep|per.key|email|\\bIP\\b path=/private/tmp/abreal/work/ab-2259468618/co-4170755525/internal" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:17Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t5.json new file mode 100644 index 0000000..6196ba3 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-01-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 3, + "cost_usd": 0.0246744, + "usage": { + "input_tokens": 6, + "output_tokens": 426, + "cache_creation_input_tokens": 4719, + "cache_read_input_tokens": 15264 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.743928625, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(limiter|ratelimit|rate_limit)", + "Grep pattern=(?i)(evict|idle|lastSeen|last used|sweep|gc) path=/private/tmp/abreal/work/ab-2259468618/co-3677836914/internal" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:23Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t1.json new file mode 100644 index 0000000..a538fcf --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go", + "turns": 3, + "cost_usd": 0.0357535, + "usage": { + "input_tokens": 6, + "output_tokens": 349, + "cache_creation_input_tokens": 7655, + "cache_read_input_tokens": 16315 + }, + "tool_calls": { + "Grep": 1, + "Read": 1 + }, + "files_read": 1, + "read_bytes": 7596, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.215028792, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(replay|assertion_?id|InResponseTo|used_?assertion|seen_?assertion)", + "Read /private/tmp/abreal/work/ab-2259468618/co-123477733/internal/saml/tracker.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:03:34Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t2.json new file mode 100644 index 0000000..4a55bef --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t2.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go", + "turns": 6, + "cost_usd": 0.0318161, + "usage": { + "input_tokens": 8, + "output_tokens": 721, + "cache_creation_input_tokens": 5577, + "cache_read_input_tokens": 22821 + }, + "tool_calls": { + "Grep": 5 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.674264791, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(replay|assertion.?id|InResponseTo|used.?assertion|seen.?id)", + "Grep pattern=(?i)saml", + "Grep pattern=(?i)assertion path=/private/tmp/abreal/work/ab-2259468618/co-1626101296/internal/saml/tracker.go", + "Grep pattern=(?i)(assertion.?id|replay)", + "Grep pattern=(?i)(assertion.?id|replay) path=/private/tmp/abreal/work/ab-2259468618/co-1626101296/internal/saml/service.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t3.json new file mode 100644 index 0000000..286d8a4 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t3.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go", + "turns": 5, + "cost_usd": 0.0358053, + "usage": { + "input_tokens": 8, + "output_tokens": 614, + "cache_creation_input_tokens": 6816, + "cache_read_input_tokens": 23853 + }, + "tool_calls": { + "Grep": 3, + "Read": 1 + }, + "files_read": 1, + "read_bytes": 2870, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.115597667, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=replay|InResponseTo|assertion.?id|used.?assertion|seen.?id", + "Grep pattern=saml", + "Read /private/tmp/abreal/work/ab-2259468618/co-2444215587/internal/saml/tracker.go", + "Grep pattern=assertion|replay path=/private/tmp/abreal/work/ab-2259468618/co-2444215587/storage" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:03Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t4.json new file mode 100644 index 0000000..d3c514c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go, internal/saml/service.go", + "turns": 4, + "cost_usd": 0.0395019, + "usage": { + "input_tokens": 6, + "output_tokens": 381, + "cache_creation_input_tokens": 8512, + "cache_read_input_tokens": 16319 + }, + "tool_calls": { + "Grep": 2, + "Read": 1 + }, + "files_read": 1, + "read_bytes": 7596, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.643879083, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(replay|assertion.?id|InResponseTo|used.?assertion|seen.?assertion)", + "Read /private/tmp/abreal/work/ab-2259468618/co-3336334775/internal/saml/tracker.go", + "Grep pattern=(?i)tracker|replay path=/private/tmp/abreal/work/ab-2259468618/co-3336334775/internal/saml/service.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:10:36Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t5.json new file mode 100644 index 0000000..866d61c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-02-t5.json @@ -0,0 +1,42 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go, internal/saml/service.go", + "turns": 7, + "cost_usd": 0.0381462, + "usage": { + "input_tokens": 10, + "output_tokens": 924, + "cache_creation_input_tokens": 6453, + "cache_read_input_tokens": 30742 + }, + "tool_calls": { + "Grep": 6 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.013364042, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(replay|assertion_?id|InResponseTo|used_?ids|seen_?ids)", + "Grep pattern=(?i)saml", + "Grep pattern=(?i)assertion.?id|replay path=/private/tmp/abreal/work/ab-2259468618/co-2783708363/internal/saml", + "Grep pattern=(?i)assertion.?id|replay path=/private/tmp/abreal/work/ab-2259468618/co-2783708363/storage", + "Grep pattern=(?i)assertion.?id|replay|Tracker path=/private/tmp/abreal/work/ab-2259468618/co-2783708363/internal/saml/tracker.go", + "Grep pattern=(?i)assertion.?id|replay path=/private/tmp/abreal/work/ab-2259468618/co-2783708363/internal/saml/service.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:05:49Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t1.json new file mode 100644 index 0000000..ab3783d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, mcpresource/jwt.go, internal/as/token_policy.go, internal/as/config.go", + "turns": 4, + "cost_usd": 0.029389300000000004, + "usage": { + "input_tokens": 6, + "output_tokens": 530, + "cache_creation_input_tokens": 5618, + "cache_read_input_tokens": 16053 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.739192041, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=EdDSA", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-1917209003" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t2.json new file mode 100644 index 0000000..968d48a --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, internal/as/config.go, internal/as/token_policy.go, internal/as/jwks.go, mcpresource/jwt.go", + "turns": 4, + "cost_usd": 0.027750500000000004, + "usage": { + "input_tokens": 6, + "output_tokens": 617, + "cache_creation_input_tokens": 5010, + "cache_read_input_tokens": 15285 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.271192667, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3262917432/internal", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3262917432/mcpresource" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:12Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t3.json new file mode 100644 index 0000000..687f41b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, internal/as/token_policy.go, internal/as/config.go, mcpresource/jwt.go, config.go", + "turns": 3, + "cost_usd": 0.0257164, + "usage": { + "input_tokens": 6, + "output_tokens": 470, + "cache_creation_input_tokens": 4869, + "cache_read_input_tokens": 15284 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.2602565, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-341532529" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:52Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t4.json new file mode 100644 index 0000000..76511a1 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t4.json @@ -0,0 +1,42 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, internal/as/token_policy.go, mcpresource/jwt.go", + "turns": 7, + "cost_usd": 0.0314825, + "usage": { + "input_tokens": 6, + "output_tokens": 979, + "cache_creation_input_tokens": 5038, + "cache_read_input_tokens": 15285 + }, + "tool_calls": { + "Grep": 6 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.01199, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3121614448/internal/jwt/multialg.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3121614448/internal/as/token_policy.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3121614448/mcpresource/jwt.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3121614448/config.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3121614448/internal/jwt/jwt.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:12Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t5.json new file mode 100644 index 0000000..2f25ee3 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-03-t5.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, mcpresource/jwt.go", + "turns": 6, + "cost_usd": 0.0289504, + "usage": { + "input_tokens": 6, + "output_tokens": 717, + "cache_creation_input_tokens": 5057, + "cache_read_input_tokens": 15404 + }, + "tool_calls": { + "Grep": 5 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.846974416, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-4130801660/internal/jwt", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-4130801660/config.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-4130801660/mcpresource/jwt.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-4130801660/internal/as/token_policy.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:05:33Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t1.json new file mode 100644 index 0000000..3995efb --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/password/lifecycle.go, internal/totp/handlers/handlers.go, lifecycle.go", + "turns": 4, + "cost_usd": 0.024787599999999996, + "usage": { + "input_tokens": 8, + "output_tokens": 526, + "cache_creation_input_tokens": 4310, + "cache_read_input_tokens": 22716 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.3216795, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=remember.?(device|browser|me)|trusted.?(device|browser)|two.?factor|otp|mfa", + "Grep pattern=trusted.?device|remember.?device|TrustedDevice|RememberDevice", + "Grep pattern=trusteddevice|TrustedDevice" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:26Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t2.json new file mode 100644 index 0000000..d9fd150 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/password/lifecycle.go, internal/totp/handlers/handlers.go", + "turns": 4, + "cost_usd": 0.024260900000000002, + "usage": { + "input_tokens": 8, + "output_tokens": 551, + "cache_creation_input_tokens": 4120, + "cache_read_input_tokens": 22549 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.328652167, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=remember.?(me|device|browser)|trusted.?(device|browser)|two.?factor|otp|2fa|mfa", + "Grep pattern=trusted.?device|remember.?(device|browser)|device.?trust", + "Grep pattern=trusteddevice|TrustedDevice" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:06:15Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t3.json new file mode 100644 index 0000000..0fd1dab --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/totp/handlers/handlers.go, internal/password/service.go, internal/password/lifecycle.go, internal/password/handlers/handlers.go", + "turns": 4, + "cost_usd": 0.0246817, + "usage": { + "input_tokens": 8, + "output_tokens": 535, + "cache_creation_input_tokens": 4263, + "cache_read_input_tokens": 22637 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.744431, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=remember.?(device|browser|me)|trusted.?(device|browser)|two.?factor|2fa|otp", + "Grep pattern=(?i)(trusted.?device|remember.?(device|browser)|device.?trust)", + "Grep pattern=(?i)trusteddevice|trusted.?device path=/private/tmp/abreal/work/ab-2259468618/co-1200086078/internal" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:18Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t4.json new file mode 100644 index 0000000..ec912f6 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/totp/handlers/handlers.go, internal/password/lifecycle.go, internal/password/service.go", + "turns": 3, + "cost_usd": 0.0186751, + "usage": { + "input_tokens": 6, + "output_tokens": 405, + "cache_creation_input_tokens": 3278, + "cache_read_input_tokens": 15011 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.582219209, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=remember.*(device|browser)|two_factor.*cookie|remember_me|trusted_device|remembe", + "Grep pattern=(?i)trusted|remember path=/private/tmp/abreal/work/ab-2259468618/co-3012551614/internal" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t5.json new file mode 100644 index 0000000..5894ca0 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-04-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/password/lifecycle.go, internal/password/service.go, internal/password/handlers/handlers.go", + "turns": 5, + "cost_usd": 0.0276803, + "usage": { + "input_tokens": 8, + "output_tokens": 675, + "cache_creation_input_tokens": 4665, + "cache_read_input_tokens": 22543 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.107201, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=remember.?(device|browser|me)|trusted.?(device|browser)|two.?factor|otp|mfa", + "Grep pattern=(?i)(trusted.?device|remember.?(device|browser)|device.?trust)", + "Grep pattern=(?i)trusteddevice|trusted.?device path=/private/tmp/abreal/work/ab-2259468618/co-1803971905/internal/password", + "Grep pattern=(?i)trusteddevice path=/private/tmp/abreal/work/ab-2259468618/co-1803971905/internal/totp/handlers/hand" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:05:16Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t1.json new file mode 100644 index 0000000..81e4cfd --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 3, + "cost_usd": 0.015563000000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 249, + "cache_creation_input_tokens": 3046, + "cache_read_input_tokens": 8810 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.6826855, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=approval", + "Glob pattern=**/*approval*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:09Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t2.json new file mode 100644 index 0000000..ad8a688 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 3, + "cost_usd": 0.015535, + "usage": { + "input_tokens": 4, + "output_tokens": 247, + "cache_creation_input_tokens": 3044, + "cache_read_input_tokens": 8810 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.022202084, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=approval", + "Glob pattern=**/*approval*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:50Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t3.json new file mode 100644 index 0000000..897388f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 2, + "cost_usd": 0.018310800000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 181, + "cache_creation_input_tokens": 3903, + "cache_read_input_tokens": 8808 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.00154425, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=approval" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:32Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t4.json new file mode 100644 index 0000000..f126536 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 3, + "cost_usd": 0.015535, + "usage": { + "input_tokens": 4, + "output_tokens": 247, + "cache_creation_input_tokens": 3044, + "cache_read_input_tokens": 8810 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.476462208, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=approval", + "Glob pattern=**/*approval*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t5.json new file mode 100644 index 0000000..2406fd5 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-05-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 3, + "cost_usd": 0.015514799999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 247, + "cache_creation_input_tokens": 3039, + "cache_read_input_tokens": 8808 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.377798917, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=approval", + "Glob pattern=**/*approval*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:59Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t1.json new file mode 100644 index 0000000..6064312 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0190214, + "usage": { + "input_tokens": 4, + "output_tokens": 123, + "cache_creation_input_tokens": 4225, + "cache_read_input_tokens": 8834 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.802943417, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t2.json new file mode 100644 index 0000000..bae2d86 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0190252, + "usage": { + "input_tokens": 4, + "output_tokens": 123, + "cache_creation_input_tokens": 4226, + "cache_read_input_tokens": 8832 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.5785415, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:19Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t3.json new file mode 100644 index 0000000..3a9d35a --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0190214, + "usage": { + "input_tokens": 4, + "output_tokens": 123, + "cache_creation_input_tokens": 4225, + "cache_read_input_tokens": 8834 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.187549208, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t4.json new file mode 100644 index 0000000..e681c8c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0190214, + "usage": { + "input_tokens": 4, + "output_tokens": 123, + "cache_creation_input_tokens": 4225, + "cache_read_input_tokens": 8834 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.144713833, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t5.json new file mode 100644 index 0000000..73c8c01 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-06-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0200154, + "usage": { + "input_tokens": 4, + "output_tokens": 118, + "cache_creation_input_tokens": 4486, + "cache_read_input_tokens": 8834 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.084414834, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:13Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t1.json new file mode 100644 index 0000000..7bc9487 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0133582, + "usage": { + "input_tokens": 4, + "output_tokens": 197, + "cache_creation_input_tokens": 2625, + "cache_read_input_tokens": 8802 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.787099292, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:05:27Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t2.json new file mode 100644 index 0000000..0eebb0f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.013434000000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 203, + "cache_creation_input_tokens": 2629, + "cache_read_input_tokens": 8800 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.034800542, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=\"[^\"]*internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:03:54Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t3.json new file mode 100644 index 0000000..e759d7a --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0134562, + "usage": { + "input_tokens": 4, + "output_tokens": 204, + "cache_creation_input_tokens": 2632, + "cache_read_input_tokens": 8802 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.211816459, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=\"[^\"]*/internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:03:23Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t4.json new file mode 100644 index 0000000..3993f9e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0134422, + "usage": { + "input_tokens": 4, + "output_tokens": 203, + "cache_creation_input_tokens": 2631, + "cache_read_input_tokens": 8802 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.040479125, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=\"[^\"]*internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:19Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t5.json new file mode 100644 index 0000000..3ecf884 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-07-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0133582, + "usage": { + "input_tokens": 4, + "output_tokens": 197, + "cache_creation_input_tokens": 2625, + "cache_read_input_tokens": 8802 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.985455375, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t1.json new file mode 100644 index 0000000..126dd2b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0131357, + "usage": { + "input_tokens": 4, + "output_tokens": 192, + "cache_creation_input_tokens": 2582, + "cache_read_input_tokens": 8797 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.399380667, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:10:21Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t2.json new file mode 100644 index 0000000..97bdd5d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0134939, + "usage": { + "input_tokens": 4, + "output_tokens": 227, + "cache_creation_input_tokens": 2584, + "cache_read_input_tokens": 8799 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.76376325, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:32Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t3.json new file mode 100644 index 0000000..9698f77 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0133439, + "usage": { + "input_tokens": 4, + "output_tokens": 212, + "cache_creation_input_tokens": 2584, + "cache_read_input_tokens": 8799 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.544640334, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:56Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t4.json new file mode 100644 index 0000000..9ce2e65 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0134157, + "usage": { + "input_tokens": 4, + "output_tokens": 220, + "cache_creation_input_tokens": 2582, + "cache_read_input_tokens": 8797 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.256428959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:45Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t5.json new file mode 100644 index 0000000..4ec1f1f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-08-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0134239, + "usage": { + "input_tokens": 4, + "output_tokens": 220, + "cache_creation_input_tokens": 2584, + "cache_read_input_tokens": 8799 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.634509334, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:09Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t1.json new file mode 100644 index 0000000..4256413 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 4, + "cost_usd": 0.0212161, + "usage": { + "input_tokens": 6, + "output_tokens": 434, + "cache_creation_input_tokens": 3832, + "cache_read_input_tokens": 15361 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.336467, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)permission", + "Grep pattern=Description path=/private/tmp/abreal/work/ab-2259468618/co-3122369932/internal/rbac/catalog.go", + "Grep pattern=^const|^\\s+Perm\\w+\\s+= path=/private/tmp/abreal/work/ab-2259468618/co-3122369932/internal/models/permissions" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:36Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t2.json new file mode 100644 index 0000000..52388a3 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 3, + "cost_usd": 0.0220941, + "usage": { + "input_tokens": 6, + "output_tokens": 331, + "cache_creation_input_tokens": 4309, + "cache_read_input_tokens": 15361 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.112716709, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)permission", + "Grep pattern=Perm\\w* *= *\"|Description path=/private/tmp/abreal/work/ab-2259468618/co-1677208033/internal" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:06:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t3.json new file mode 100644 index 0000000..70e3da5 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t3.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 5, + "cost_usd": 0.026664999999999998, + "usage": { + "input_tokens": 8, + "output_tokens": 609, + "cache_creation_input_tokens": 4566, + "cache_read_input_tokens": 22950 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.457656209, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)permission", + "Grep pattern=Description|Perm[A-Z]\\w+\\s*= path=/private/tmp/abreal/work/ab-2259468618/co-3857493092/internal/rbac/catalog.go", + "Grep pattern=Perm[A-Z]\\w+\\s*= path=/private/tmp/abreal/work/ab-2259468618/co-3857493092/internal/models/permissions", + "Grep pattern=PermissionBillingRead\\b path=/private/tmp/abreal/work/ab-2259468618/co-3857493092/internal/models" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:30Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t4.json new file mode 100644 index 0000000..b099be5 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 5, + "cost_usd": 0.025434699999999998, + "usage": { + "input_tokens": 8, + "output_tokens": 609, + "cache_creation_input_tokens": 4266, + "cache_read_input_tokens": 22647 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.784203041, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)permission", + "Grep pattern=Description|Perm[A-Z]\\w+\\s*= path=/private/tmp/abreal/work/ab-2259468618/co-1296084940/internal/rbac/catalog.go", + "Grep pattern=Perm[A-Z]\\w+\\s*= path=/private/tmp/abreal/work/ab-2259468618/co-1296084940/internal/models/permissions", + "Grep pattern=PermissionBillingRead\\b path=/private/tmp/abreal/work/ab-2259468618/co-1296084940/internal/models" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:10:27Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t5.json new file mode 100644 index 0000000..a9b0b7e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-09-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 4, + "cost_usd": 0.023942099999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 449, + "cache_creation_input_tokens": 4476, + "cache_read_input_tokens": 15361 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.3340175, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)permission", + "Grep pattern=Permission|Description path=/private/tmp/abreal/work/ab-2259468618/co-1358275759/internal/rbac/catalog.go", + "Grep pattern=const|= \" path=/private/tmp/abreal/work/ab-2259468618/co-1358275759/internal/models/permissions" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:40Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t1.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t1.json new file mode 100644 index 0000000..f7bb03c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 2, + "cost_usd": 0.0161814, + "usage": { + "input_tokens": 4, + "output_tokens": 177, + "cache_creation_input_tokens": 3380, + "cache_read_input_tokens": 8834 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.253959292, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(otp|one.?time).*sender|sender.*(otp|one.?time)|interface\\s+\\w*Sender|class\\" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:47Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t2.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t2.json new file mode 100644 index 0000000..ce2f67c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 3, + "cost_usd": 0.0191077, + "usage": { + "input_tokens": 6, + "output_tokens": 276, + "cache_creation_input_tokens": 3717, + "cache_read_input_tokens": 14677 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.279081209, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(otp|one.?time).*(sender|send)|(sender|send).*(otp|one.?time)", + "Grep pattern=Sender path=otp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:06:08Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t3.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t3.json new file mode 100644 index 0000000..fdd264c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 4, + "cost_usd": 0.021097099999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 402, + "cache_creation_input_tokens": 3895, + "cache_read_input_tokens": 14851 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.099903667, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)(otp|one.?time).*(sender|send)|(sender|send).*(otp|one.?time)", + "Grep pattern=(?i)(interface|trait|protocol|abstract class)\\s+\\w*Sender", + "Grep pattern=Sender path=/private/tmp/abreal/work/ab-2259468618/co-2187594415/otp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t4.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t4.json new file mode 100644 index 0000000..c30296c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 4, + "cost_usd": 0.0216566, + "usage": { + "input_tokens": 6, + "output_tokens": 442, + "cache_creation_input_tokens": 3935, + "cache_read_input_tokens": 14846 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.608775167, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(otp|one.?time|code).*sender|sender.*(otp|code)", + "Grep pattern=(?i)(interface|trait|protocol|abstract class)\\s+\\w*Sender", + "Grep pattern=Sender path=/private/tmp/abreal/work/ab-2259468618/co-3265964864/otp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:03:48Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t5.json b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t5.json new file mode 100644 index 0000000..1c4428a --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/control/tg2-10-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 3, + "cost_usd": 0.0158114, + "usage": { + "input_tokens": 4, + "output_tokens": 310, + "cache_creation_input_tokens": 2955, + "cache_read_input_tokens": 8834 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.015324417, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(otp|one.?time).*(sender|send)|(sender|send).*(otp|one.?time)", + "Grep pattern=(?i)class\\s+\\w*Sender|interface\\s+\\w*Sender|type\\s+\\w*Sender|Sender\\s*\\(Protocol" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:03:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t1.json new file mode 100644 index 0000000..25cb62b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 4, + "cost_usd": 0.024585300000000004, + "usage": { + "input_tokens": 8, + "output_tokens": 449, + "cache_creation_input_tokens": 4349, + "cache_read_input_tokens": 26833 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.322672958, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=rate\\.NewLimiter|rate\\.Limiter", + "Grep pattern=evict|lastSeen|cleanup|delete\\( path=internal/ratelimit", + "Grep pattern=evict|lastSeen|cleanup|delete\\(|^type |^func New path=internal/ratelimit" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:06:32Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t2.json new file mode 100644 index 0000000..d4cac63 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 4, + "cost_usd": 0.0273452, + "usage": { + "input_tokens": 8, + "output_tokens": 453, + "cache_creation_input_tokens": 5030, + "cache_read_input_tokens": 26792 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.34784125, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=rate\\.NewLimiter|rate\\.Limiter", + "Grep pattern=lastSeen|evict|cleanup|delete\\( path=/private/tmp/abreal/work/ab-2259468618/co-920304433/internal/ratelimit", + "Grep pattern=lastSeen|evict|cleanup|delete\\(|^type |^func path=/private/tmp/abreal/work/ab-2259468618/co-920304433/internal/ratelimit" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:47Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t3.json new file mode 100644 index 0000000..c31cbb7 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t3.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 5, + "cost_usd": 0.0276767, + "usage": { + "input_tokens": 8, + "output_tokens": 639, + "cache_creation_input_tokens": 4647, + "cache_read_input_tokens": 26827 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.003701417, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=rate\\.NewLimiter|rate\\.Limiter", + "Grep pattern=evict|cleanup|lastSeen|last_seen path=/private/tmp/abreal/work/ab-2259468618/co-4081468180/internal/ratelimit", + "Grep pattern=evict|cleanup|lastSeen|last_seen|^func |^type path=/private/tmp/abreal/work/ab-2259468618/co-4081468180/internal/ratelimit/window.g", + "Grep pattern=evict|cleanup|lastSeen|last_seen|^func |^type path=/private/tmp/abreal/work/ab-2259468618/co-4081468180/internal/ratelimit/ratelimi" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:39Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t4.json new file mode 100644 index 0000000..f0ed069 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 4, + "cost_usd": 0.023905700000000002, + "usage": { + "input_tokens": 8, + "output_tokens": 412, + "cache_creation_input_tokens": 4273, + "cache_read_input_tokens": 26777 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.137645667, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=rate\\.NewLimiter|rate\\.Limiter", + "Grep pattern=evict|lastSeen|cleanup|delete\\( path=internal/ratelimit", + "Grep pattern=evict|lastSeen|cleanup|delete\\(|^type |^// Package path=internal/ratelimit" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:31Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t5.json new file mode 100644 index 0000000..8724eec --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-01-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-01", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/ratelimit/ratelimit.go", + "turns": 5, + "cost_usd": 0.026681700000000003, + "usage": { + "input_tokens": 8, + "output_tokens": 694, + "cache_creation_input_tokens": 4260, + "cache_read_input_tokens": 26857 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.493308125, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=rate\\.NewLimiter|rate\\.Limiter", + "Grep pattern=evict|lastSeen|cleanup|janitor path=internal/ratelimit", + "Grep pattern=evict|lastSeen|cleanup|janitor|rate\\.Limiter|map\\[string\\] path=internal/ratelimit/window.go", + "Grep pattern=evict|lastSeen|rate\\.Limiter|map\\[string\\] path=internal/ratelimit/ratelimit.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:45Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t1.json new file mode 100644 index 0000000..b50ddc4 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t1.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go, internal/saml/service.go, kv/kv.go, kv/memory.go", + "turns": 6, + "cost_usd": 0.034745200000000004, + "usage": { + "input_tokens": 12, + "output_tokens": 742, + "cache_creation_input_tokens": 5666, + "cache_read_input_tokens": 46372 + }, + "tool_calls": { + "Grep": 5 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.094086916, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)replay|assertion.?id|AssertionID", + "Grep pattern=(?i)saml.*(replay|assertion)|(replay|assertion).*saml", + "Grep pattern=(?i)replay|seen|consume path=/private/tmp/abreal/work/ab-2259468618/co-2670544232/internal/saml/service.go", + "Grep pattern=SeenAssertion\\(", + "Grep pattern=^func|^type|kv\\.|Store path=/private/tmp/abreal/work/ab-2259468618/co-2670544232/internal/saml/tracker.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:43Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t2.json new file mode 100644 index 0000000..410a6a7 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t2.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go, internal/saml/service.go", + "turns": 6, + "cost_usd": 0.0361465, + "usage": { + "input_tokens": 10, + "output_tokens": 760, + "cache_creation_input_tokens": 6199, + "cache_read_input_tokens": 37305 + }, + "tool_calls": { + "Grep": 5 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 10.451821333, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)replay|assertion.?id|InResponseTo", + "Grep pattern=(?i)saml.*(replay|assertion)|(replay|assertion).*saml", + "Grep pattern=(?i)replay|assertion.?id path=/private/tmp/abreal/work/ab-2259468618/co-1368620907/internal/saml/service.go", + "Grep pattern=(?i)saml.*(replay|consume|seen)|(replay|consume|seen).*saml", + "Grep pattern=Tracker path=/private/tmp/abreal/work/ab-2259468618/co-1368620907/internal/saml" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:06:49Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t3.json new file mode 100644 index 0000000..531bf5c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go, internal/saml/service.go", + "turns": 4, + "cost_usd": 0.034023, + "usage": { + "input_tokens": 8, + "output_tokens": 492, + "cache_creation_input_tokens": 6542, + "cache_read_input_tokens": 29190 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.3358065, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)replay|assertion.?id|AssertionID", + "Grep pattern=(?i)replay|tracker|assertion path=/private/tmp/abreal/work/ab-2259468618/co-3888001543/internal/saml", + "Grep pattern=(?i)tracker path=/private/tmp/abreal/work/ab-2259468618/co-3888001543/internal/saml" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:36Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t4.json new file mode 100644 index 0000000..07e17bd --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go, internal/saml/service.go", + "turns": 3, + "cost_usd": 0.029080800000000004, + "usage": { + "input_tokens": 6, + "output_tokens": 316, + "cache_creation_input_tokens": 5971, + "cache_read_input_tokens": 20248 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.359093583, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)replay", + "Grep pattern=(?i)replay|tracker path=/private/tmp/abreal/work/ab-2259468618/co-1494391697/internal/saml" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:49Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t5.json new file mode 100644 index 0000000..0c4a127 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-02-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-02", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/saml/tracker.go", + "turns": 4, + "cost_usd": 0.030145799999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 482, + "cache_creation_input_tokens": 5821, + "cache_read_input_tokens": 20298 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.746984583, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)replay|assertion.?id|AssertionID", + "Grep pattern=(?i)assertion|replay path=/private/tmp/abreal/work/ab-2259468618/co-1396745310/internal/saml/tracker.go", + "Grep pattern=(?i)tracker|replay path=/private/tmp/abreal/work/ab-2259468618/co-1396745310/internal/saml/service.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:06:24Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t1.json new file mode 100644 index 0000000..a54fd1f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t1.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, mcpresource/jwt.go", + "turns": 5, + "cost_usd": 0.0288152, + "usage": { + "input_tokens": 6, + "output_tokens": 671, + "cache_creation_input_tokens": 5030, + "cache_read_input_tokens": 19732 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.001428333, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-1031943516/internal/jwt", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-1031943516/internal/as/token_policy.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-1031943516/mcpresource/jwt.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:10:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t2.json new file mode 100644 index 0000000..5f1645c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, internal/as/token_policy.go, mcpresource/jwt.go, internal/as/config.go, config.go", + "turns": 5, + "cost_usd": 0.0286513, + "usage": { + "input_tokens": 6, + "output_tokens": 643, + "cache_creation_input_tokens": 5057, + "cache_read_input_tokens": 19813 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.917571083, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3466287400/internal/jwt", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3466287400/internal/as/config.go", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-3466287400/config.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:24Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t3.json new file mode 100644 index 0000000..85473e2 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, internal/as/jwks.go, internal/as/token_policy.go, internal/as/config.go, config.go, mcpresource/jwt.go", + "turns": 4, + "cost_usd": 0.032344, + "usage": { + "input_tokens": 6, + "output_tokens": 594, + "cache_creation_input_tokens": 6088, + "cache_read_input_tokens": 20400 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.468273917, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=EdDSA", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-2438530561" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:10:43Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t4.json new file mode 100644 index 0000000..d5cb011 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, mcpresource/jwt.go, internal/as/token_policy.go, internal/as/config.go, internal/as/jwks.go", + "turns": 4, + "cost_usd": 0.0329066, + "usage": { + "input_tokens": 6, + "output_tokens": 574, + "cache_creation_input_tokens": 6274, + "cache_read_input_tokens": 20586 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.252940792, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=EdDSA", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-690292505" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:09Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t5.json new file mode 100644 index 0000000..1b866de --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-03-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-03", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/jwt/multialg.go, internal/as/jwks.go, internal/as/token_policy.go, internal/as/config.go, mcpresource/jwt.go", + "turns": 3, + "cost_usd": 0.030078, + "usage": { + "input_tokens": 6, + "output_tokens": 418, + "cache_creation_input_tokens": 5979, + "cache_read_input_tokens": 19700 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.262302375, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=ES256|RS256", + "Grep pattern=ES256|RS256 path=/private/tmp/abreal/work/ab-2259468618/co-10056763" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:08Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t1.json new file mode 100644 index 0000000..3b68f82 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/password/service.go, internal/totp/handlers/handlers.go", + "turns": 4, + "cost_usd": 0.022963400000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 508, + "cache_creation_input_tokens": 3979, + "cache_read_input_tokens": 19554 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.133552709, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)remember.?(device|browser|me).*(cookie|sign)|trusted.?(device|browser)|remem", + "Grep pattern=(?i)trusteddevice|trusted.?device path=/private/tmp/abreal/work/ab-2259468618/co-298849054/internal/password/service.go", + "Grep pattern=(?i)Sign|HMAC|Revoke|cookie path=/private/tmp/abreal/work/ab-2259468618/co-298849054/internal/trusteddevice/servi" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:10:12Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t2.json new file mode 100644 index 0000000..e54a9f3 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/password/lifecycle.go, internal/password/service.go, internal/password/handlers/handlers.go, internal/totp/handlers/handlers.go", + "turns": 4, + "cost_usd": 0.024001500000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 528, + "cache_creation_input_tokens": 4191, + "cache_read_input_tokens": 19455 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 10.945706125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)remember.?(me|device|browser)|trusted.?(device|browser)", + "Grep pattern=(?i)trusteddevice|trusted.?device path=/private/tmp/abreal/work/ab-2259468618/co-1103601855/internal/password", + "Grep pattern=(?i)trusteddevice path=/private/tmp/abreal/work/ab-2259468618/co-1103601855/internal/totp/handlers/hand" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:39Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t3.json new file mode 100644 index 0000000..f9494ad --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/totp/handlers/handlers.go, internal/password/lifecycle.go, internal/password/service.go", + "turns": 3, + "cost_usd": 0.020326200000000003, + "usage": { + "input_tokens": 6, + "output_tokens": 307, + "cache_creation_input_tokens": 3820, + "cache_read_input_tokens": 19642 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.099916209, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)remember.?(device|browser|me)|trusted.?(device|browser)", + "Grep pattern=(?i)trusteddevice|trusted.?device" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:34Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t4.json new file mode 100644 index 0000000..c5ac7de --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/password/service.go, internal/totp/handlers/handlers.go", + "turns": 3, + "cost_usd": 0.021625699999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 428, + "cache_creation_input_tokens": 3846, + "cache_read_input_tokens": 19497 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.027203833, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=remember.?(me|device|browser)|trusted.?(device|browser)|remember_?2fa|skip.?(2fa", + "Grep pattern=trusteddevice|TrustedDevice" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:11Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t5.json new file mode 100644 index 0000000..cfe0cad --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-04-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-04", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "internal/trusteddevice/service.go, internal/password/lifecycle.go, internal/password/service.go, internal/password/handlers/handlers.go, internal/totp/handlers/handlers.go", + "turns": 4, + "cost_usd": 0.0237615, + "usage": { + "input_tokens": 6, + "output_tokens": 502, + "cache_creation_input_tokens": 4196, + "cache_read_input_tokens": 19455 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.902004625, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)remember.?(me|device|browser)|trusted.?(device|browser)", + "Grep pattern=(?i)trusteddevice|trusted.?device path=/private/tmp/abreal/work/ab-2259468618/co-3119746856/internal/password", + "Grep pattern=(?i)hmac|sign path=/private/tmp/abreal/work/ab-2259468618/co-3119746856/internal/trusteddevice/serv" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t1.json new file mode 100644 index 0000000..71b6345 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t1.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 4, + "cost_usd": 0.0201794, + "usage": { + "input_tokens": 8, + "output_tokens": 359, + "cache_creation_input_tokens": 3472, + "cache_read_input_tokens": 26854 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.104537, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)approval", + "Glob pattern=storage/sqlite/*toolapproval*", + "Grep pattern=(?i)tool_?approval path=storage/sqlite" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:57Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t2.json new file mode 100644 index 0000000..f9d69d5 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 4, + "cost_usd": 0.020318299999999997, + "usage": { + "input_tokens": 8, + "output_tokens": 368, + "cache_creation_input_tokens": 3484, + "cache_read_input_tokens": 26863 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.7691529169999995, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)approval", + "Glob pattern=storage/sqlite/*toolapproval*", + "Grep pattern=(?i)tool_?approval path=storage/sqlite" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:05Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t3.json new file mode 100644 index 0000000..e9f2685 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t3.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 4, + "cost_usd": 0.020329800000000002, + "usage": { + "input_tokens": 8, + "output_tokens": 382, + "cache_creation_input_tokens": 3451, + "cache_read_input_tokens": 26898 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.31217075, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=approval", + "Glob pattern=storage/sqlite/*toolapproval*", + "Grep pattern=ToolApproval path=storage/sqlite" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:10:03Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t4.json new file mode 100644 index 0000000..5a7e95f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 4, + "cost_usd": 0.020511299999999996, + "usage": { + "input_tokens": 8, + "output_tokens": 395, + "cache_creation_input_tokens": 3464, + "cache_read_input_tokens": 26893 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.025919791, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=approval", + "Glob pattern=storage/sqlite/*toolapproval*", + "Grep pattern=ToolApproval path=storage/sqlite" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:21Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t5.json new file mode 100644 index 0000000..dd6cd7b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-05-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-05", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "storage/memory/memory_toolapprovals.go, storage/sqlite/tool_approvals.go, storage/postgres/postgres_toolapprovals.go", + "turns": 4, + "cost_usd": 0.0201794, + "usage": { + "input_tokens": 8, + "output_tokens": 359, + "cache_creation_input_tokens": 3472, + "cache_read_input_tokens": 26854 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.343609584, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)approval", + "Glob pattern=storage/sqlite/*toolapproval*", + "Grep pattern=(?i)tool_?approval path=storage/sqlite" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t1.json new file mode 100644 index 0000000..5b62917 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0193826, + "usage": { + "input_tokens": 4, + "output_tokens": 147, + "cache_creation_input_tokens": 4184, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.000152292, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:04Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t2.json new file mode 100644 index 0000000..2071a88 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0193826, + "usage": { + "input_tokens": 4, + "output_tokens": 147, + "cache_creation_input_tokens": 4184, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.89993075, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:47Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t3.json new file mode 100644 index 0000000..b168308 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0200646, + "usage": { + "input_tokens": 4, + "output_tokens": 142, + "cache_creation_input_tokens": 4367, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.291752958, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:05Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t4.json new file mode 100644 index 0000000..5454a83 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0193826, + "usage": { + "input_tokens": 4, + "output_tokens": 147, + "cache_creation_input_tokens": 4184, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.375894, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t5.json new file mode 100644 index 0000000..90b0bd2 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-06-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-06", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/totp/handlers/handlers.go, internal/totp/service.go, storage/sqlite/totp.go", + "turns": 2, + "cost_usd": 0.0193826, + "usage": { + "input_tokens": 4, + "output_tokens": 147, + "cache_creation_input_tokens": 4184, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.088051791, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)totp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:31Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t1.json new file mode 100644 index 0000000..f10f6ca --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0159674, + "usage": { + "input_tokens": 4, + "output_tokens": 227, + "cache_creation_input_tokens": 3131, + "cache_read_input_tokens": 11654 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.900985, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=\"[^\"]*internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:14:21Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t2.json new file mode 100644 index 0000000..a3d1229 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.015951, + "usage": { + "input_tokens": 4, + "output_tokens": 227, + "cache_creation_input_tokens": 3127, + "cache_read_input_tokens": 11650 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.353935875, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=\"[^\"]*internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t3.json new file mode 100644 index 0000000..8d57d43 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0159674, + "usage": { + "input_tokens": 4, + "output_tokens": 227, + "cache_creation_input_tokens": 3131, + "cache_read_input_tokens": 11654 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.803891458, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=\"[^\"]*internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:58Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t4.json new file mode 100644 index 0000000..aa3b9f5 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0159674, + "usage": { + "input_tokens": 4, + "output_tokens": 227, + "cache_creation_input_tokens": 3131, + "cache_read_input_tokens": 11654 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.00722625, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=\"[^\"]*internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t5.json new file mode 100644 index 0000000..7f1eda4 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-07-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-07", + "kind": "impact", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "signals/urlcheck.go, signals/transmitter.go, signals/receiver.go, policy/hook/httpguard.go, internal/cimd/safefetch.go, internal/as/oidc_logout.go, internal/as/jwks_fetcher.go", + "turns": 2, + "cost_usd": 0.0159674, + "usage": { + "input_tokens": 4, + "output_tokens": 227, + "cache_creation_input_tokens": 3131, + "cache_read_input_tokens": 11654 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.751811792, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=\"[^\"]*internal/safehttp\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:30Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t1.json new file mode 100644 index 0000000..0714e25 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0156431, + "usage": { + "input_tokens": 4, + "output_tokens": 223, + "cache_creation_input_tokens": 3060, + "cache_read_input_tokens": 11651 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.657217666, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:27Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t2.json new file mode 100644 index 0000000..ead5675 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.015623100000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 221, + "cache_creation_input_tokens": 3060, + "cache_read_input_tokens": 11651 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.568845417, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:13:18Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t3.json new file mode 100644 index 0000000..5d13ada --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0156267, + "usage": { + "input_tokens": 4, + "output_tokens": 223, + "cache_creation_input_tokens": 3056, + "cache_read_input_tokens": 11647 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.340194583, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:08:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t4.json new file mode 100644 index 0000000..3ddc125 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0155131, + "usage": { + "input_tokens": 4, + "output_tokens": 210, + "cache_creation_input_tokens": 3060, + "cache_read_input_tokens": 11651 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.474887541, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:03:29Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t5.json new file mode 100644 index 0000000..756c9d7 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-08-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth-go", + "task": "tg2-08", + "kind": "impact", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "theauth.go, internal/scim/handlers/handlers.go, internal/saml/service.go, internal/password/service.go, internal/magiclink/service.go, forwarders.go, email/noop.go, cmd/theauth-go/init.go", + "turns": 2, + "cost_usd": 0.0155531, + "usage": { + "input_tokens": 4, + "output_tokens": 214, + "cache_creation_input_tokens": 3060, + "cache_read_input_tokens": 11651 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.104762208, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=internal/emailnorm\"" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:09:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t1.json new file mode 100644 index 0000000..8273015 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t1.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 4, + "cost_usd": 0.028692300000000004, + "usage": { + "input_tokens": 8, + "output_tokens": 434, + "cache_creation_input_tokens": 5389, + "cache_read_input_tokens": 27803 + }, + "tool_calls": { + "Bash": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.737614666, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)permission", + "Grep pattern=Permission\\w* *= *\"|Description:.*[Pp]ermission|\"users\\.(read|manage)|PermUsers", + "Bash cd /private/tmp/abreal/work/ab-2259468618/co-301930174; head -30 internal/models/permissions.go; grep -n \"\" internal/rbac/catalog.go | head -40" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:06:41Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t2.json new file mode 100644 index 0000000..b1b1ddb --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 5, + "cost_usd": 0.030167, + "usage": { + "input_tokens": 8, + "output_tokens": 599, + "cache_creation_input_tokens": 5321, + "cache_read_input_tokens": 28770 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.553978625, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)permission.*(webhook|description)|PermissionManage|\"manage_", + "Grep pattern=Permission|webhook path=/private/tmp/abreal/work/ab-2259468618/co-1809654519/internal/rbac/catalog.go", + "Grep pattern=Permission path=/private/tmp/abreal/work/ab-2259468618/co-1809654519/enterprise.go", + "Grep pattern=PermissionBillingRead\\s*=" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t3.json new file mode 100644 index 0000000..c9f2336 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 3, + "cost_usd": 0.024409399999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 297, + "cache_creation_input_tokens": 4874, + "cache_read_input_tokens": 19314 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.217178416, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)permission.*(webhook|description)|PermissionName|Permission[A-Z]\\w* += ", + "Bash cd /private/tmp/abreal/work/ab-2259468618/co-3461480798; head -40 internal/models/permissions.go; echo ----; head -40 internal/rbac/catalog.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:05:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t4.json new file mode 100644 index 0000000..45d9c83 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 5, + "cost_usd": 0.024738999999999997, + "usage": { + "input_tokens": 6, + "output_tokens": 591, + "cache_creation_input_tokens": 4218, + "cache_read_input_tokens": 19450 + }, + "tool_calls": { + "Grep": 4 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.933734167, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)permission.*description|Description.*permission", + "Grep pattern=(?i)PermissionManage|PermManage|Permission\\w+\\s*=\\s*\"", + "Grep pattern=Description path=/private/tmp/abreal/work/ab-2259468618/co-418832424/internal/rbac/catalog.go", + "Grep pattern=Permission\\w+\\s*=\\s*\" path=/private/tmp/abreal/work/ab-2259468618/co-418832424/internal/models/permissions." + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:05Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t5.json new file mode 100644 index 0000000..29119d4 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-09-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-09", + "kind": "add", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "internal/models/permissions.go, internal/rbac/catalog.go", + "turns": 4, + "cost_usd": 0.024557799999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 484, + "cache_creation_input_tokens": 4446, + "cache_read_input_tokens": 19218 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.046696, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)permission.*description|Permission\\w* *= *\"", + "Grep pattern=Permission|Description path=/private/tmp/abreal/work/ab-2259468618/co-667848139/internal/rbac/catalog.go", + "Grep pattern=Permission|Description path=/private/tmp/abreal/work/ab-2259468618/co-667848139/internal/models/permissions." + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:07:15Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t1.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t1.json new file mode 100644 index 0000000..154a19f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t1.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 5, + "cost_usd": 0.0244756, + "usage": { + "input_tokens": 6, + "output_tokens": 486, + "cache_creation_input_tokens": 4408, + "cache_read_input_tokens": 19716 + }, + "tool_calls": { + "Grep": 3, + "Read": 1 + }, + "files_read": 1, + "read_bytes": 697, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 9.610166625, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)(type|interface)\\s+\\w*(Sender|Deliver)\\w*\\s+interface", + "Grep pattern=(?i)otp|one-time|onetime", + "Grep pattern=\\) Send\\( path=/private/tmp/abreal/work/ab-2259468618/co-2324660082/otp", + "Read /private/tmp/abreal/work/ab-2259468618/co-2324660082/otp/otp.go" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:15Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t2.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t2.json new file mode 100644 index 0000000..b362387 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 3, + "cost_usd": 0.0225125, + "usage": { + "input_tokens": 6, + "output_tokens": 281, + "cache_creation_input_tokens": 4438, + "cache_read_input_tokens": 19385 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.72150625, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)type\\s+\\w*(sender|otp|code)\\w*\\s+interface", + "Grep pattern=otp\\.Sender|\\) Send\\(|Sender path=/private/tmp/abreal/work/ab-2259468618/co-3167116335/otp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:11:24Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t3.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t3.json new file mode 100644 index 0000000..cacad50 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 3, + "cost_usd": 0.0186226, + "usage": { + "input_tokens": 4, + "output_tokens": 269, + "cache_creation_input_tokens": 3689, + "cache_read_input_tokens": 11686 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.109579584, + "seven_day_utilization": 0.68, + "call_log": [ + "Grep pattern=(?i)type\\s+\\w*(sender|otp|code)\\w*\\s+interface", + "Grep pattern=(?i)sender" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:04:36Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t4.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t4.json new file mode 100644 index 0000000..496b08e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 3, + "cost_usd": 0.0217354, + "usage": { + "input_tokens": 6, + "output_tokens": 268, + "cache_creation_input_tokens": 4279, + "cache_read_input_tokens": 19274 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.740866042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)type\\s+\\w*(sender|otp|notifier)\\w*\\s+interface", + "Grep pattern=Sender path=/private/tmp/abreal/work/ab-2259468618/co-3152803066/otp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:12:30Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t5.json b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t5.json new file mode 100644 index 0000000..49e7b96 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth-go/stacklit-quiet/tg2-10-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth-go", + "task": "tg2-10", + "kind": "add", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "otp/otp.go, otp/senders.go", + "turns": 3, + "cost_usd": 0.0195013, + "usage": { + "input_tokens": 6, + "output_tokens": 271, + "cache_creation_input_tokens": 3713, + "cache_read_input_tokens": 19273 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.499927125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)type\\s+\\w*(sender|otp|sms)\\w*\\s+interface", + "Bash cd /private/tmp/abreal/work/ab-2259468618/co-3118007302; sed -n 45,75p otp/otp.go; grep -rn \"otp.Sender\\|) Send(\" --include=*.go otp | head -20; ls otp" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:15:26Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-01-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-01-t1.json new file mode 100644 index 0000000..01c7553 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-01-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 3, + "cost_usd": 0.0167944, + "usage": { + "input_tokens": 6, + "output_tokens": 310, + "cache_creation_input_tokens": 3049, + "cache_read_input_tokens": 14864 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.949769, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=haveibeenpwned|pwnedpasswords|hibp|range/", + "Grep pattern=pwnedpasswords|api\\.pwned" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:26:35Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-01-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-01-t2.json new file mode 100644 index 0000000..868579b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-01-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 2, + "cost_usd": 0.014329999999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 214, + "cache_creation_input_tokens": 2825, + "cache_read_input_tokens": 8820 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.742745459, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|range/" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:28:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-01-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-01-t3.json new file mode 100644 index 0000000..73c400e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-01-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 2, + "cost_usd": 0.01456, + "usage": { + "input_tokens": 4, + "output_tokens": 237, + "cache_creation_input_tokens": 2825, + "cache_read_input_tokens": 8820 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.352165458, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|range/" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:31:54Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-01-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-01-t4.json new file mode 100644 index 0000000..eb3b79d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-01-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 2, + "cost_usd": 0.014315999999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 211, + "cache_creation_input_tokens": 2829, + "cache_read_input_tokens": 8820 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.20150275, + "seven_day_utilization": 0.7, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|HIBP|range/" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:41:49Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-01-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-01-t5.json new file mode 100644 index 0000000..e79fe6f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-01-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 3, + "cost_usd": 0.016752799999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 307, + "cache_creation_input_tokens": 3046, + "cache_read_input_tokens": 14868 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.677655959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|range/", + "Grep pattern=pwnedpasswords\\.com|api\\.pwned" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:30:53Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-02-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-02-t1.json new file mode 100644 index 0000000..518de2e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-02-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 3, + "cost_usd": 0.0157149, + "usage": { + "input_tokens": 6, + "output_tokens": 244, + "cache_creation_input_tokens": 2948, + "cache_read_input_tokens": 14709 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.219216875, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|turnstile|hcaptcha|recaptcha", + "Grep pattern=siteverify" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:26:06Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-02-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-02-t2.json new file mode 100644 index 0000000..97158d6 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-02-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 2, + "cost_usd": 0.0128404, + "usage": { + "input_tokens": 4, + "output_tokens": 163, + "cache_creation_input_tokens": 2580, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.729193125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|challenges\\.cloudflare\\.com/turnstile|hcaptcha\\.com/siteverify|recapt" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:33:04Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-02-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-02-t3.json new file mode 100644 index 0000000..dff50ac --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-02-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 2, + "cost_usd": 0.0128322, + "usage": { + "input_tokens": 4, + "output_tokens": 163, + "cache_creation_input_tokens": 2578, + "cache_read_input_tokens": 8822 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.417038209, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|challenges\\.cloudflare\\.com/turnstile|hcaptcha\\.com/siteverify|recapt" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:40:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-02-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-02-t4.json new file mode 100644 index 0000000..e6bfe97 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-02-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 2, + "cost_usd": 0.0128404, + "usage": { + "input_tokens": 4, + "output_tokens": 163, + "cache_creation_input_tokens": 2580, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.121186875, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|challenges\\.cloudflare\\.com/turnstile|hcaptcha\\.com/siteverify|recapt" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:35:39Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-02-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-02-t5.json new file mode 100644 index 0000000..b01fdad --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-02-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 3, + "cost_usd": 0.0151909, + "usage": { + "input_tokens": 6, + "output_tokens": 242, + "cache_creation_input_tokens": 2822, + "cache_read_input_tokens": 14709 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.618434416, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|turnstile|hcaptcha|recaptcha", + "Grep pattern=siteverify path=packages" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:29:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-03-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-03-t1.json new file mode 100644 index 0000000..5c580e0 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-03-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.0175673, + "usage": { + "input_tokens": 6, + "output_tokens": 355, + "cache_creation_input_tokens": 3130, + "cache_read_input_tokens": 14853 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 10.049017042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)cbor", + "Grep pattern=(?i)max|depth path=/private/tmp/abreal/work/ab-2412612584/co-2085333810/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:30:16Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-03-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-03-t2.json new file mode 100644 index 0000000..a86a1ee --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-03-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.017483300000000004, + "usage": { + "input_tokens": 6, + "output_tokens": 349, + "cache_creation_input_tokens": 3124, + "cache_read_input_tokens": 14853 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.623579292, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)cbor", + "Grep pattern=(?i)max|depth path=/private/tmp/abreal/work/ab-2412612584/co-1317013737/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:40:18Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-03-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-03-t3.json new file mode 100644 index 0000000..b88e7aa --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-03-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.0170502, + "usage": { + "input_tokens": 6, + "output_tokens": 326, + "cache_creation_input_tokens": 3074, + "cache_read_input_tokens": 14822 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.78988075, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)cbor", + "Grep pattern=(?i)max|depth path=/private/tmp/abreal/work/ab-2412612584/co-236445301/packages/core/src/auth/cbor." + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:41:15Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-03-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-03-t4.json new file mode 100644 index 0000000..d4de1c2 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-03-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.0171169, + "usage": { + "input_tokens": 6, + "output_tokens": 323, + "cache_creation_input_tokens": 3098, + "cache_read_input_tokens": 14829 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.877261958, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)cbor", + "Grep pattern=(?i)max|depth path=/private/tmp/abreal/work/ab-2412612584/co-1038070105/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:37:45Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-03-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-03-t5.json new file mode 100644 index 0000000..05c6093 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-03-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.0172502, + "usage": { + "input_tokens": 6, + "output_tokens": 334, + "cache_creation_input_tokens": 3104, + "cache_read_input_tokens": 14822 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.539443959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)cbor", + "Grep pattern=(?i)max|depth path=/private/tmp/abreal/work/ab-2412612584/co-839841281/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:28:40Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-04-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-04-t1.json new file mode 100644 index 0000000..84d8960 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-04-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 3, + "cost_usd": 0.018134400000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 244, + "cache_creation_input_tokens": 3701, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.255343416, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=anonymi[sz]e|erasure", + "Glob pattern=**/core/**/*{erasure,erase,anonym,delete}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:40:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-04-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-04-t2.json new file mode 100644 index 0000000..1934fa8 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-04-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 3, + "cost_usd": 0.0200042, + "usage": { + "input_tokens": 4, + "output_tokens": 261, + "cache_creation_input_tokens": 4126, + "cache_read_input_tokens": 8822 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.56354125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]e|anonymi[sz]ation", + "Glob pattern=**/core/**/*{erasure,anonym,delete,checkpoint,progress}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:29:18Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-04-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-04-t3.json new file mode 100644 index 0000000..195c75c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-04-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 3, + "cost_usd": 0.0175004, + "usage": { + "input_tokens": 4, + "output_tokens": 267, + "cache_creation_input_tokens": 3485, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.666331958, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]e", + "Glob pattern=**/core/**/*{erasure,anonym,delete}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:33:36Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-04-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-04-t4.json new file mode 100644 index 0000000..cedbaa4 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-04-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 3, + "cost_usd": 0.0192202, + "usage": { + "input_tokens": 4, + "output_tokens": 275, + "cache_creation_input_tokens": 3895, + "cache_read_input_tokens": 8822 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.765862291, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]", + "Glob pattern=**/core/**/*{erasure,anonym,delete,checkpoint,progress}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:38:56Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-04-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-04-t5.json new file mode 100644 index 0000000..f7de828 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-04-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 3, + "cost_usd": 0.019168400000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 247, + "cache_creation_input_tokens": 3952, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.359177875, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=anonymi[sz]e|erasure|erase", + "Glob pattern=**/core/**/*{erasure,erase,anonym,delete}*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:28:30Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-05-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-05-t1.json new file mode 100644 index 0000000..c369fd6 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-05-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.016319100000000003, + "usage": { + "input_tokens": 4, + "output_tokens": 263, + "cache_creation_input_tokens": 3199, + "cache_read_input_tokens": 8851 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.488842083, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|device_authorization", + "Grep pattern=token cache|TokenCache|token_cache|tokenCache" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:39:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-05-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-05-t2.json new file mode 100644 index 0000000..ef1d0b3 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-05-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 4, + "cost_usd": 0.0231114, + "usage": { + "input_tokens": 6, + "output_tokens": 341, + "cache_creation_input_tokens": 4527, + "cache_read_input_tokens": 15814 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.719818959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|device_authorization", + "Grep pattern=token.*cache|TokenCache|credentials", + "Glob pattern=packages/cli/src/**/*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:37:19Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-05-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-05-t3.json new file mode 100644 index 0000000..f31f615 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-05-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.0172931, + "usage": { + "input_tokens": 4, + "output_tokens": 302, + "cache_creation_input_tokens": 3345, + "cache_read_input_tokens": 8851 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.374074125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|device_authorization", + "Grep pattern=token.?cache|TokenCache" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:39:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-05-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-05-t4.json new file mode 100644 index 0000000..e03fe1c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-05-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 4, + "cost_usd": 0.0185138, + "usage": { + "input_tokens": 6, + "output_tokens": 357, + "cache_creation_input_tokens": 3358, + "cache_read_input_tokens": 14998 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.626616125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|device_authorization", + "Glob pattern=packages/cli/src/auth/*", + "Grep pattern=device-auth|cache|DeviceAuth path=/private/tmp/abreal/work/ab-2412612584/co-1371507846/packages/cli/src/auth/devic" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:26:55Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-05-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-05-t5.json new file mode 100644 index 0000000..dddd65a --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-05-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.0166391, + "usage": { + "input_tokens": 4, + "output_tokens": 305, + "cache_creation_input_tokens": 3174, + "cache_read_input_tokens": 8851 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.711727125, + "seven_day_utilization": 0.7, + "call_log": [ + "Grep pattern=device_code|device/code|device_authorization", + "Grep pattern=token cache|TokenCache|token_cache|tokenCache" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:41:22Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-06-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-06-t1.json new file mode 100644 index 0000000..c6a7570 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-06-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.013002300000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 2648, + "cache_read_input_tokens": 8823 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.366334834, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic-link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:32:04Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-06-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-06-t2.json new file mode 100644 index 0000000..9934d44 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-06-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.013002300000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 2648, + "cache_read_input_tokens": 8823 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.164438375, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic*link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:36:43Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-06-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-06-t3.json new file mode 100644 index 0000000..10fb3ab --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-06-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.013002300000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 2648, + "cache_read_input_tokens": 8823 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.2390695, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic*link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:39:07Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-06-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-06-t4.json new file mode 100644 index 0000000..5f0bc90 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-06-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.013002300000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 2648, + "cache_read_input_tokens": 8823 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.081964042, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic-link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:34:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-06-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-06-t5.json new file mode 100644 index 0000000..5418653 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-06-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.013002300000000001, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 2648, + "cache_read_input_tokens": 8823 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.499448625, + "seven_day_utilization": 0.7, + "call_log": [ + "Glob pattern=**/*magic-link*", + "Glob pattern=**/*magic*link*/**" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:41:44Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-07-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-07-t1.json new file mode 100644 index 0000000..ea9b5f0 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-07-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/user/user.ts, packages/core/src/auth/email-verification.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/magic-link.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/auth/oauth/plugin.ts", + "turns": 2, + "cost_usd": 0.015958, + "usage": { + "input_tokens": 4, + "output_tokens": 304, + "cache_creation_input_tokens": 3007, + "cache_read_input_tokens": 8820 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.562891292, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:28:35Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-07-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-07-t2.json new file mode 100644 index 0000000..7ded17f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-07-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/email-verification.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/auth/oauth/plugin.ts, packages/core/src/user/user.ts, packages/core/src/hooks/database-hooks.ts", + "turns": 2, + "cost_usd": 0.015958, + "usage": { + "input_tokens": 4, + "output_tokens": 304, + "cache_creation_input_tokens": 3007, + "cache_read_input_tokens": 8820 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.454962667, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:26:26Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-07-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-07-t3.json new file mode 100644 index 0000000..eb32df0 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-07-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/email-verification.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/password-reset.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/user/user.ts, packages/core/src/auth/oauth/plugin.ts", + "turns": 2, + "cost_usd": 0.015937800000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 304, + "cache_creation_input_tokens": 3002, + "cache_read_input_tokens": 8818 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.581831125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:48Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-07-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-07-t4.json new file mode 100644 index 0000000..802c26d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-07-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/email-verification.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/oauth/plugin.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/user/user.ts", + "turns": 2, + "cost_usd": 0.015958, + "usage": { + "input_tokens": 4, + "output_tokens": 304, + "cache_creation_input_tokens": 3007, + "cache_read_input_tokens": 8820 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.64680425, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:32:08Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-07-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-07-t5.json new file mode 100644 index 0000000..7ec97fe --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-07-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/user/user.ts, packages/core/src/auth/email-verification.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/oauth/plugin.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/organization-plugin.ts", + "turns": 2, + "cost_usd": 0.016248, + "usage": { + "input_tokens": 4, + "output_tokens": 333, + "cache_creation_input_tokens": 3007, + "cache_read_input_tokens": 8820 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.999034666, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:36:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-08-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-08-t1.json new file mode 100644 index 0000000..8c4681c --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-08-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/username-http.ts, packages/core/src/auth/username.ts, packages/core/src/auth/email-password.ts", + "turns": 2, + "cost_usd": 0.0153884, + "usage": { + "input_tokens": 4, + "output_tokens": 187, + "cache_creation_input_tokens": 3157, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.670632208, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:35:28Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-08-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-08-t2.json new file mode 100644 index 0000000..010ba07 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-08-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/username-http.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/username.ts", + "turns": 2, + "cost_usd": 0.015444399999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 191, + "cache_creation_input_tokens": 3161, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.141445542, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:33:56Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-08-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-08-t3.json new file mode 100644 index 0000000..3302890 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-08-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/username-http.ts, packages/core/src/auth/username.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/email-password.ts", + "turns": 2, + "cost_usd": 0.0153884, + "usage": { + "input_tokens": 4, + "output_tokens": 187, + "cache_creation_input_tokens": 3157, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.183218375, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:38:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-08-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-08-t4.json new file mode 100644 index 0000000..e6d050e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-08-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/username-http.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/username.ts, packages/core/src/auth/email-password.ts", + "turns": 2, + "cost_usd": 0.015444399999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 191, + "cache_creation_input_tokens": 3161, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 3.979130208, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:38:31Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-08-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-08-t5.json new file mode 100644 index 0000000..f692a4e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-08-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/username.ts, packages/core/src/auth/username-http.ts", + "turns": 2, + "cost_usd": 0.015444399999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 191, + "cache_creation_input_tokens": 3161, + "cache_read_input_tokens": 8824 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 4.12481725, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:32:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-09-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-09-t1.json new file mode 100644 index 0000000..5448fbd --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-09-t1.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/\u003cnew-plugin\u003e-plugin.ts, packages/core/src/auth/index.ts", + "turns": 4, + "cost_usd": 0.0206346, + "usage": { + "input_tokens": 6, + "output_tokens": 280, + "cache_creation_input_tokens": 4088, + "cache_read_input_tokens": 14706 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.206265541, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*", + "Glob pattern=**/*magicLink*", + "Grep pattern=magic-link path=/private/tmp/abreal/work/ab-2412612584/co-3196886562/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:39:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-09-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-09-t2.json new file mode 100644 index 0000000..4521ae2 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-09-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/\u003cname\u003e-plugin.ts, packages/core/src/auth/index.ts", + "turns": 4, + "cost_usd": 0.020594599999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 276, + "cache_creation_input_tokens": 4088, + "cache_read_input_tokens": 14706 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.20712, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/magic-link*/**", + "Glob pattern=**/magic-link*", + "Grep pattern=magic-link path=/private/tmp/abreal/work/ab-2412612584/co-1336448387/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:29:06Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-09-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-09-t3.json new file mode 100644 index 0000000..66dc528 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-09-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/index.ts", + "turns": 3, + "cost_usd": 0.0192402, + "usage": { + "input_tokens": 6, + "output_tokens": 196, + "cache_creation_input_tokens": 3952, + "cache_read_input_tokens": 14602 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.726290541, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*", + "Grep pattern=magic-link path=packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:37:01Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-09-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-09-t4.json new file mode 100644 index 0000000..409228a --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-09-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/index.ts", + "turns": 4, + "cost_usd": 0.0206046, + "usage": { + "input_tokens": 6, + "output_tokens": 277, + "cache_creation_input_tokens": 4088, + "cache_read_input_tokens": 14706 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.149509166, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*", + "Glob pattern=**/*magicLink*", + "Grep pattern=magic-link path=/private/tmp/abreal/work/ab-2412612584/co-3766350267/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:35:33Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-09-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-09-t5.json new file mode 100644 index 0000000..601eb8b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-09-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/index.ts", + "turns": 4, + "cost_usd": 0.0204184, + "usage": { + "input_tokens": 6, + "output_tokens": 264, + "cache_creation_input_tokens": 4074, + "cache_read_input_tokens": 14704 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 8.700243917, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*", + "Glob pattern=**/plugins/index.ts", + "Grep pattern=magic-link path=packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:40:54Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-10-t1.json b/bench/agent-ab/results-round3/theauth/control/ta-10-t1.json new file mode 100644 index 0000000..974689d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-10-t1.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 1, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 5, + "cost_usd": 0.020237, + "usage": { + "input_tokens": 6, + "output_tokens": 499, + "cache_creation_input_tokens": 3436, + "cache_read_input_tokens": 14910 + }, + "tool_calls": { + "Glob": 1, + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.4154750830000005, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=locales", + "Glob pattern=**/locales/**", + "Grep pattern=locales|export path=/private/tmp/abreal/work/ab-2412612584/co-1812374950/packages/core/src/i18n/inde", + "Grep pattern=locales|import path=/private/tmp/abreal/work/ab-2412612584/co-1812374950/packages/core/src/i18n/i18n" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:34:24Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-10-t2.json b/bench/agent-ab/results-round3/theauth/control/ta-10-t2.json new file mode 100644 index 0000000..b2fa4c8 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-10-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 2, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 4, + "cost_usd": 0.018211, + "usage": { + "input_tokens": 6, + "output_tokens": 354, + "cache_creation_input_tokens": 3292, + "cache_read_input_tokens": 14910 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 5.239154959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=locales", + "Glob pattern=**/locales/**", + "Grep pattern=locales path=/private/tmp/abreal/work/ab-2412612584/co-2954894946/packages/core/src/i18n" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:29:12Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-10-t3.json b/bench/agent-ab/results-round3/theauth/control/ta-10-t3.json new file mode 100644 index 0000000..5e35deb --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-10-t3.json @@ -0,0 +1,41 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 3, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 5, + "cost_usd": 0.0198748, + "usage": { + "input_tokens": 6, + "output_tokens": 434, + "cache_creation_input_tokens": 3509, + "cache_read_input_tokens": 14868 + }, + "tool_calls": { + "Glob": 2, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.049276042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=locales", + "Glob pattern=**/locales/**", + "Grep pattern=locales path=/private/tmp/abreal/work/ab-2412612584/co-626745669/packages/core/src/i18n", + "Glob pattern=packages/core/src/i18n/**" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:37:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-10-t4.json b/bench/agent-ab/results-round3/theauth/control/ta-10-t4.json new file mode 100644 index 0000000..014aa0f --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-10-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 4, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 4, + "cost_usd": 0.0182006, + "usage": { + "input_tokens": 6, + "output_tokens": 353, + "cache_creation_input_tokens": 3292, + "cache_read_input_tokens": 14906 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 6.187496792, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=locales", + "Glob pattern=**/locales/**", + "Grep pattern=locales path=/private/tmp/abreal/work/ab-2412612584/co-825478626/packages/core/src/i18n" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:26:20Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/control/ta-10-t5.json b/bench/agent-ab/results-round3/theauth/control/ta-10-t5.json new file mode 100644 index 0000000..6ae47a8 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/control/ta-10-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 5, + "arm": "control", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 4, + "cost_usd": 0.018211, + "usage": { + "input_tokens": 6, + "output_tokens": 354, + "cache_creation_input_tokens": 3292, + "cache_read_input_tokens": 14910 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 0, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 7.179706875, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/locales/**", + "Grep pattern=locales", + "Grep pattern=locales path=/private/tmp/abreal/work/ab-2412612584/co-1063955747/packages/core/src/i18n" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:26:13Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t1.json new file mode 100644 index 0000000..6056ecc --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 3, + "cost_usd": 0.0195138, + "usage": { + "input_tokens": 6, + "output_tokens": 334, + "cache_creation_input_tokens": 3555, + "cache_read_input_tokens": 19418 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.763652083, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|range/", + "Grep pattern=pwnedpasswords|range/ path=/private/tmp/abreal/work/ab-2412612584/co-2995390207/packages" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:29:34Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t2.json new file mode 100644 index 0000000..5ff22fd --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 3, + "cost_usd": 0.0189098, + "usage": { + "input_tokens": 6, + "output_tokens": 290, + "cache_creation_input_tokens": 3514, + "cache_read_input_tokens": 19418 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.5715305, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|range/", + "Grep pattern=api\\.pwnedpasswords|range/" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:31:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t3.json new file mode 100644 index 0000000..1878fa0 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 3, + "cost_usd": 0.019662, + "usage": { + "input_tokens": 6, + "output_tokens": 339, + "cache_creation_input_tokens": 3579, + "cache_read_input_tokens": 19440 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.692115041, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|api\\.pwned", + "Grep pattern=pwnedpasswords|range/ path=/private/tmp/abreal/work/ab-2412612584/co-3294600536/packages" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:31:12Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t4.json new file mode 100644 index 0000000..02e9208 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 2, + "cost_usd": 0.0172756, + "usage": { + "input_tokens": 4, + "output_tokens": 268, + "cache_creation_input_tokens": 3355, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.374760292, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|range/" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:31:00Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t5.json new file mode 100644 index 0000000..8ee3db8 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-01-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-01", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/hibp.ts", + "turns": 2, + "cost_usd": 0.0161672, + "usage": { + "input_tokens": 4, + "output_tokens": 168, + "cache_creation_input_tokens": 3328, + "cache_read_input_tokens": 11672 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.575926125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=pwnedpasswords|haveibeenpwned|hibp|range/" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:32:52Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t1.json new file mode 100644 index 0000000..110f1e8 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 3, + "cost_usd": 0.0185022, + "usage": { + "input_tokens": 6, + "output_tokens": 267, + "cache_creation_input_tokens": 3473, + "cache_read_input_tokens": 19282 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.5420275, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|turnstile|hcaptcha|recaptcha", + "Grep pattern=siteverify path=packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:39:50Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t2.json new file mode 100644 index 0000000..2ea676d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 2, + "cost_usd": 0.014962, + "usage": { + "input_tokens": 4, + "output_tokens": 157, + "cache_creation_input_tokens": 3054, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.806101959, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|recaptcha/api|challenges\\.cloudflare\\.com|hcaptcha\\.com" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:30:04Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t3.json new file mode 100644 index 0000000..0e7caa9 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 2, + "cost_usd": 0.014836, + "usage": { + "input_tokens": 4, + "output_tokens": 148, + "cache_creation_input_tokens": 3045, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.084590542, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|recaptcha/api|hcaptcha\\.com/siteverify" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:38:19Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t4.json new file mode 100644 index 0000000..4baebb5 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 2, + "cost_usd": 0.0148056, + "usage": { + "input_tokens": 4, + "output_tokens": 147, + "cache_creation_input_tokens": 3040, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 16.091857458, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|recaptcha/api|hcaptcha\\.com/siteverify" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:38:35Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t5.json new file mode 100644 index 0000000..56bd8bc --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-02-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-02", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/captcha.ts", + "turns": 2, + "cost_usd": 0.0152256, + "usage": { + "input_tokens": 4, + "output_tokens": 177, + "cache_creation_input_tokens": 3070, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.365201291, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=siteverify|recaptcha/api|hcaptcha\\.com/siteverify" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:29:22Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t1.json new file mode 100644 index 0000000..5299618 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.0201999, + "usage": { + "input_tokens": 6, + "output_tokens": 416, + "cache_creation_input_tokens": 3524, + "cache_read_input_tokens": 19319 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 14.292861084, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cbor", + "Grep pattern=depth|MAX_|max path=/private/tmp/abreal/work/ab-2412612584/co-1192499795/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:35:59Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t2.json new file mode 100644 index 0000000..477b7cc --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.0194919, + "usage": { + "input_tokens": 6, + "output_tokens": 324, + "cache_creation_input_tokens": 3575, + "cache_read_input_tokens": 19399 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 14.542748083, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)cbor", + "Grep pattern=(?i)max|depth path=/private/tmp/abreal/work/ab-2412612584/co-2742370451/packages/core/src/auth/cbor" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:39:22Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t3.json new file mode 100644 index 0000000..ce46775 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 4, + "cost_usd": 0.0283196, + "usage": { + "input_tokens": 8, + "output_tokens": 525, + "cache_creation_input_tokens": 5083, + "cache_read_input_tokens": 27216 + }, + "tool_calls": { + "Grep": 3 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 15.690810167, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=(?i)cbor", + "Grep pattern=(?i)max|depth path=/private/tmp/abreal/work/ab-2412612584/co-1960159702/packages/core/src", + "Grep pattern=(?i)max|depth|passkey|webauthn path=/private/tmp/abreal/work/ab-2412612584/co-1960159702/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:35:44Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t4.json new file mode 100644 index 0000000..046affb --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.020039400000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 362, + "cache_creation_input_tokens": 3617, + "cache_read_input_tokens": 19394 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.766644916, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cbor", + "Grep pattern=depth|MAX_ path=/private/tmp/abreal/work/ab-2412612584/co-3876890744/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:33:23Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t5.json new file mode 100644 index 0000000..f06a993 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-03-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-03", + "kind": "locate", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": -1, + "answer": "packages/core/src/auth/cbor.ts", + "turns": 3, + "cost_usd": 0.0196334, + "usage": { + "input_tokens": 6, + "output_tokens": 341, + "cache_creation_input_tokens": 3568, + "cache_read_input_tokens": 19394 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.47228425, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=cbor", + "Grep pattern=MAX|depth path=/private/tmp/abreal/work/ab-2412612584/co-1227254433/packages/core/src/auth/cbor" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:30:26Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t1.json new file mode 100644 index 0000000..9be86b4 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 2, + "cost_usd": 0.018163600000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 226, + "cache_creation_input_tokens": 3682, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.583581, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]e" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:35:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t2.json new file mode 100644 index 0000000..4490683 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 2, + "cost_usd": 0.018164, + "usage": { + "input_tokens": 4, + "output_tokens": 226, + "cache_creation_input_tokens": 3682, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.383059625, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]e" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:37:07Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t3.json new file mode 100644 index 0000000..daf3bdd --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 2, + "cost_usd": 0.018164, + "usage": { + "input_tokens": 4, + "output_tokens": 226, + "cache_creation_input_tokens": 3682, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.542338083, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]e" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:40:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t4.json new file mode 100644 index 0000000..8732c1d --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 2, + "cost_usd": 0.017906000000000002, + "usage": { + "input_tokens": 4, + "output_tokens": 227, + "cache_creation_input_tokens": 3615, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.479476584, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]e" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:36:31Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t5.json new file mode 100644 index 0000000..7bfb275 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-04-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-04", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/gdpr-delete.ts, packages/core/src/auth/gdpr-anon.ts, packages/core/src/auth/gdpr-progress.ts", + "turns": 2, + "cost_usd": 0.018164, + "usage": { + "input_tokens": 4, + "output_tokens": 226, + "cache_creation_input_tokens": 3682, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 10.998094459, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=erasure|anonymi[sz]e" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:40:02Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t1.json new file mode 100644 index 0000000..b709fac --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.019684, + "usage": { + "input_tokens": 6, + "output_tokens": 261, + "cache_creation_input_tokens": 3773, + "cache_read_input_tokens": 19700 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 19.847092042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|deviceCode|device/code", + "Bash ls packages/cli/src/auth/; grep -n \"import\" packages/cli/src/auth/device-flow.ts | head -20" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:28:05Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t2.json new file mode 100644 index 0000000..69b678a --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t2.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.0204941, + "usage": { + "input_tokens": 6, + "output_tokens": 313, + "cache_creation_input_tokens": 3845, + "cache_read_input_tokens": 19721 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.501574917, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|deviceCode", + "Bash ls /private/tmp/abreal/work/ab-2412612584/co-2021856237/packages/cli/src/auth; grep -n \"import\" /private/tmp/abreal/work/ab-2412612584/co-2021856237/packages/cl" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:37:32Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t3.json new file mode 100644 index 0000000..6d1e0cf --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t3.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.01967, + "usage": { + "input_tokens": 6, + "output_tokens": 260, + "cache_creation_input_tokens": 3772, + "cache_read_input_tokens": 19700 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 15.740327042, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|deviceCode", + "Bash ls packages/cli/src/auth; grep -n \"import\" packages/cli/src/auth/device-flow.ts | head -20" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:34:59Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t4.json new file mode 100644 index 0000000..b96fe7b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t4.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.020409999999999998, + "usage": { + "input_tokens": 6, + "output_tokens": 312, + "cache_creation_input_tokens": 3827, + "cache_read_input_tokens": 19700 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.9680555, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|deviceCode", + "Bash ls /private/tmp/abreal/work/ab-2412612584/co-2909572122/packages/cli/src/auth; grep -n \"import\" /private/tmp/abreal/work/ab-2412612584/co-2909572122/packages/cl" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:37:51Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t5.json new file mode 100644 index 0000000..6ef46ce --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-05-t5.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-05", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/cli/src/auth/device-flow.ts, packages/cli/src/auth/token-cache.ts, packages/core/src/auth/device-auth.ts", + "turns": 3, + "cost_usd": 0.020412, + "usage": { + "input_tokens": 6, + "output_tokens": 313, + "cache_creation_input_tokens": 3825, + "cache_read_input_tokens": 19700 + }, + "tool_calls": { + "Bash": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 14.686219416, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=device_code|device/code|deviceCode", + "Bash ls /private/tmp/abreal/work/ab-2412612584/co-1103228515/packages/cli/src/auth/; grep -n \"import\" /private/tmp/abreal/work/ab-2412612584/co-1103228515/packages/c" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:32:37Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t1.json new file mode 100644 index 0000000..5baa2d7 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t1.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.0152079, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 3128, + "cache_read_input_tokens": 11679 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.7850155, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic*link*/**", + "Glob pattern=**/*magic-link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:33:10Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t2.json new file mode 100644 index 0000000..cc71630 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t2.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.0152079, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 3128, + "cache_read_input_tokens": 11679 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.539124375, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic*link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:41:03Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t3.json new file mode 100644 index 0000000..c947d60 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.0152079, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 3128, + "cache_read_input_tokens": 11679 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.338684834, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic-link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:36:19Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t4.json new file mode 100644 index 0000000..4a069e6 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.015215699999999999, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 3130, + "cache_read_input_tokens": 11677 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 17.037795875, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic-link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:34:06Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t5.json new file mode 100644 index 0000000..7e2ceef --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-06-t5.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-06", + "kind": "trace", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts", + "turns": 3, + "cost_usd": 0.0152079, + "usage": { + "input_tokens": 4, + "output_tokens": 152, + "cache_creation_input_tokens": 3128, + "cache_read_input_tokens": 11679 + }, + "tool_calls": { + "Glob": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.703115834, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*/**", + "Glob pattern=**/*magic*link*" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:32:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t1.json new file mode 100644 index 0000000..b064387 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/user/user.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/auth/email-verification.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/oauth/plugin.ts", + "turns": 2, + "cost_usd": 0.0185832, + "usage": { + "input_tokens": 4, + "output_tokens": 330, + "cache_creation_input_tokens": 3527, + "cache_read_input_tokens": 11672 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.589156958, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:40:25Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t2.json new file mode 100644 index 0000000..aa738e2 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/hooks/database-hooks.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-verification.ts, packages/core/src/user/user.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/oauth/plugin.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/email-otp.ts", + "turns": 2, + "cost_usd": 0.0186954, + "usage": { + "input_tokens": 4, + "output_tokens": 330, + "cache_creation_input_tokens": 3555, + "cache_read_input_tokens": 11674 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 20.457766875, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:27:44Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t3.json new file mode 100644 index 0000000..14cc90b --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t3.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/user/user.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/oauth/plugin.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-verification.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/email-otp-plugin.ts", + "turns": 3, + "cost_usd": 0.021224500000000004, + "usage": { + "input_tokens": 6, + "output_tokens": 469, + "cache_creation_input_tokens": 3648, + "cache_read_input_tokens": 19305 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 14.912673125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email", + "Grep pattern=from .*normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:34:43Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t4.json new file mode 100644 index 0000000..61ead11 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t4.json @@ -0,0 +1,38 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/user/user.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/one-tap.ts, packages/core/src/auth/oauth/plugin.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-verification.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/email-otp-plugin.ts", + "turns": 3, + "cost_usd": 0.021775700000000002, + "usage": { + "input_tokens": 6, + "output_tokens": 509, + "cache_creation_input_tokens": 3685, + "cache_read_input_tokens": 19337 + }, + "tool_calls": { + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 15.838330125, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email", + "Grep pattern=from .*normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:31:38Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t5.json new file mode 100644 index 0000000..84eb829 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-07-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-07", + "kind": "impact", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/email-verification.ts, packages/core/src/auth/magic-link.ts, packages/core/src/auth/oauth/plugin.ts, packages/core/src/auth/one-tap.ts, packages/core/src/hooks/database-hooks.ts, packages/core/src/auth/password-reset.ts, packages/core/src/auth/email-otp-plugin.ts, packages/core/src/auth/email-otp.ts, packages/core/src/auth/organization-plugin.ts, packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/email-password.ts, packages/core/src/user/user.ts", + "turns": 2, + "cost_usd": 0.0186712, + "usage": { + "input_tokens": 4, + "output_tokens": 330, + "cache_creation_input_tokens": 3549, + "cache_read_input_tokens": 11672 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.369107583, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=normalize-email" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:27:02Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t1.json new file mode 100644 index 0000000..756bfa4 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t1.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/username-http.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/index.ts, packages/core/src/auth/username.ts, packages/core/src/auth/email-password-http.ts", + "turns": 2, + "cost_usd": 0.01795, + "usage": { + "input_tokens": 4, + "output_tokens": 187, + "cache_creation_input_tokens": 3726, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.146571709, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:33:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t2.json new file mode 100644 index 0000000..2af7943 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t2.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/username.ts, packages/core/src/auth/username-http.ts", + "turns": 2, + "cost_usd": 0.01795, + "usage": { + "input_tokens": 4, + "output_tokens": 187, + "cache_creation_input_tokens": 3726, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 10.82670175, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:39:11Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t3.json new file mode 100644 index 0000000..6d0e572 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t3.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/username.ts, packages/core/src/auth/username-http.ts", + "turns": 2, + "cost_usd": 0.0179436, + "usage": { + "input_tokens": 4, + "output_tokens": 212, + "cache_creation_input_tokens": 3662, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 11.666494291, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:32:13Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t4.json new file mode 100644 index 0000000..3a3b306 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t4.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/username.ts, packages/core/src/auth/username-http.ts", + "turns": 2, + "cost_usd": 0.0178856, + "usage": { + "input_tokens": 4, + "output_tokens": 209, + "cache_creation_input_tokens": 3655, + "cache_read_input_tokens": 11676 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.708789333, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:25:53Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t5.json new file mode 100644 index 0000000..614f835 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-08-t5.json @@ -0,0 +1,37 @@ +{ + "repo": "theauth", + "task": "ta-08", + "kind": "impact", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/index.ts, packages/core/src/auth/email-password-http.ts, packages/core/src/auth/email-password.ts, packages/core/src/auth/username.ts, packages/core/src/auth/username-http.ts", + "turns": 2, + "cost_usd": 0.017944, + "usage": { + "input_tokens": 4, + "output_tokens": 212, + "cache_creation_input_tokens": 3662, + "cache_read_input_tokens": 11680 + }, + "tool_calls": { + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 12.746409041, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=login-lockout" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:34:30Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t1.json new file mode 100644 index 0000000..0d273e0 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t1.json @@ -0,0 +1,42 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/index.ts", + "turns": 5, + "cost_usd": 0.024628699999999996, + "usage": { + "input_tokens": 8, + "output_tokens": 407, + "cache_creation_input_tokens": 4451, + "cache_read_input_tokens": 27387 + }, + "tool_calls": { + "Bash": 1, + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 16.070529625, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/magic-link*/**", + "Glob pattern=**/plugins/index.ts", + "Grep pattern=magic-?link", + "Bash cd packages/core; ls src; find src -ipath '*magic*'; grep -rn -i \"magic\" src --include=index.ts | head; grep -n -i magic package.json" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:29:48Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t2.json new file mode 100644 index 0000000..1279523 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t2.json @@ -0,0 +1,42 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/magic-link-plugin.ts, packages/core/src/auth/index.ts", + "turns": 5, + "cost_usd": 0.0248475, + "usage": { + "input_tokens": 8, + "output_tokens": 425, + "cache_creation_input_tokens": 4461, + "cache_read_input_tokens": 27375 + }, + "tool_calls": { + "Bash": 1, + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 16.518323416, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/magic-link/**", + "Glob pattern=**/plugins/index.ts", + "Grep pattern=magic-?link", + "Bash cd packages/core \u0026\u0026 ls src; ls src/plugins 2\u003e/dev/null | head -50; grep -rn \"magic-link\" src --include=index.ts | head; grep -n \"magic-link\\|magicLink\" package." + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:27:27Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t3.json new file mode 100644 index 0000000..35bf374 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t3.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/\u003cyour-plugin\u003e-plugin.ts, packages/core/src/auth/index.ts", + "turns": 4, + "cost_usd": 0.022717600000000004, + "usage": { + "input_tokens": 6, + "output_tokens": 259, + "cache_creation_input_tokens": 4548, + "cache_read_input_tokens": 19236 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.539822166, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/*magic-link*", + "Glob pattern=packages/core/src/plugins/index.ts", + "Grep pattern=magic-link path=packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:28:46Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t4.json new file mode 100644 index 0000000..0cb1199 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/\u003cname\u003e-plugin.ts, packages/core/src/auth/index.ts", + "turns": 4, + "cost_usd": 0.0229589, + "usage": { + "input_tokens": 6, + "output_tokens": 276, + "cache_creation_input_tokens": 4566, + "cache_read_input_tokens": 19229 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 14.477296458, + "seven_day_utilization": 0.7, + "call_log": [ + "Glob pattern=**/magic-link*/**", + "Glob pattern=**/magic-link*", + "Grep pattern=magic-link path=/private/tmp/abreal/work/ab-2412612584/co-2068074315/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:41:29Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t5.json new file mode 100644 index 0000000..458714e --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-09-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-09", + "kind": "add", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/auth/\u003cyour-plugin\u003e-plugin.ts, packages/core/src/auth/index.ts", + "turns": 4, + "cost_usd": 0.022998900000000003, + "usage": { + "input_tokens": 6, + "output_tokens": 280, + "cache_creation_input_tokens": 4566, + "cache_read_input_tokens": 19229 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.08306125, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/magic-link*/**", + "Glob pattern=**/magic-link*", + "Grep pattern=magic-link path=/private/tmp/abreal/work/ab-2412612584/co-3667819532/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:30:40Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t1.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t1.json new file mode 100644 index 0000000..5eef7d1 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t1.json @@ -0,0 +1,39 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 1, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 3, + "cost_usd": 0.0169669, + "usage": { + "input_tokens": 4, + "output_tokens": 253, + "cache_creation_input_tokens": 3315, + "cache_read_input_tokens": 11689 + }, + "tool_calls": { + "Glob": 1, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.117451916, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=packages/core/src/**/locale*/**", + "Grep pattern=export .*locales|locales\\b.*from path=/private/tmp/abreal/work/ab-2412612584/co-1000351067/packages/core" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:26:42Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t2.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t2.json new file mode 100644 index 0000000..36d2f32 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t2.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 2, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 4, + "cost_usd": 0.020989, + "usage": { + "input_tokens": 6, + "output_tokens": 383, + "cache_creation_input_tokens": 3801, + "cache_read_input_tokens": 19430 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.664503542, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=locales", + "Glob pattern=**/locales/**/*.ts", + "Grep pattern=locales path=/private/tmp/abreal/work/ab-2412612584/co-689077904/packages/core/src/i18n" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:36:47Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t3.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t3.json new file mode 100644 index 0000000..98f6c44 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t3.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 3, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 4, + "cost_usd": 0.0190519, + "usage": { + "input_tokens": 6, + "output_tokens": 292, + "cache_creation_input_tokens": 3545, + "cache_read_input_tokens": 19399 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 13.535092792, + "seven_day_utilization": 0.69, + "call_log": [ + "Glob pattern=**/locales/**/*.ts", + "Glob pattern=packages/core/**/locale*/**", + "Grep pattern=locales/(de|fr) path=/private/tmp/abreal/work/ab-2412612584/co-4231298588/packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:27:14Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t4.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t4.json new file mode 100644 index 0000000..40541e2 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t4.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 4, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 4, + "cost_usd": 0.0226928, + "usage": { + "input_tokens": 6, + "output_tokens": 504, + "cache_creation_input_tokens": 3922, + "cache_read_input_tokens": 19528 + }, + "tool_calls": { + "Glob": 1, + "Grep": 2 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 14.596800916, + "seven_day_utilization": 0.69, + "call_log": [ + "Grep pattern=locales path=/private/tmp/abreal/work/ab-2412612584/co-1084366696", + "Glob pattern=packages/core/**/locales/** path=/private/tmp/abreal/work/ab-2412612584/co-1084366696", + "Grep pattern=locales path=/private/tmp/abreal/work/ab-2412612584/co-1084366696/packages/core/src/i18n" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:38:05Z" +} \ No newline at end of file diff --git a/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t5.json b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t5.json new file mode 100644 index 0000000..98d1b20 --- /dev/null +++ b/bench/agent-ab/results-round3/theauth/stacklit-quiet/ta-10-t5.json @@ -0,0 +1,40 @@ +{ + "repo": "theauth", + "task": "ta-10", + "kind": "add", + "trial": 5, + "arm": "stacklit-quiet", + "model": "claude-sonnet-5-5", + "correct": true, + "recall": 1, + "answer": "packages/core/src/i18n/index.ts, packages/core/src/i18n/locales/", + "turns": 4, + "cost_usd": 0.0182363, + "usage": { + "input_tokens": 6, + "output_tokens": 266, + "cache_creation_input_tokens": 3409, + "cache_read_input_tokens": 19283 + }, + "tool_calls": { + "Glob": 2, + "Grep": 1 + }, + "files_read": 0, + "read_bytes": 0, + "stacklit_calls": 0, + "hook_events": 2, + "crawl_guard_fired": false, + "crawl_redirects": 0, + "permission_denials": 0, + "result_subtype": "success", + "seconds": 15.631313583, + "seven_day_utilization": 0.7, + "call_log": [ + "Glob pattern=packages/core/**/locales/**", + "Glob pattern=**/locales/index.ts", + "Grep pattern=locales/(de|fr) path=packages/core/src" + ], + "stacklit_bin": "0870521fa963", + "at": "2026-11-09T04:41:54Z" +} \ No newline at end of file diff --git a/bench/agent-ab/round3/ledger.jsonl b/bench/agent-ab/round3/ledger.jsonl new file mode 100644 index 0000000..d0918de --- /dev/null +++ b/bench/agent-ab/round3/ledger.jsonl @@ -0,0 +1,300 @@ +{"arm":"control","at":"2026-11-09T04:03:23Z","cost_usd":0.0134562,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.0134562,"task":"tg2-07","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:03:29Z","cost_usd":0.0155131,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.0289693,"task":"tg2-08","trial":4} +{"arm":"control","at":"2026-11-09T04:03:34Z","cost_usd":0.0357535,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.0647228,"task":"tg2-02","trial":1} +{"arm":"control","at":"2026-11-09T04:03:42Z","cost_usd":0.0158114,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.0805342,"task":"tg2-10","trial":5} +{"arm":"control","at":"2026-11-09T04:03:48Z","cost_usd":0.0216566,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.1021908,"task":"tg2-10","trial":4} +{"arm":"control","at":"2026-11-09T04:03:54Z","cost_usd":0.013434000000000001,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.1156248,"task":"tg2-07","trial":2} +{"arm":"control","at":"2026-11-09T04:04:03Z","cost_usd":0.029527400000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.1451522,"task":"tg2-01","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:04:15Z","cost_usd":0.0244756,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.1696278,"task":"tg2-10","trial":1} +{"arm":"control","at":"2026-11-09T04:04:25Z","cost_usd":0.015535,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.1851628,"task":"tg2-05","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:04:30Z","cost_usd":0.0159674,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.20113019999999998,"task":"tg2-07","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:04:36Z","cost_usd":0.0186226,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.21975279999999997,"task":"tg2-10","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:04:42Z","cost_usd":0.030167,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.24991979999999997,"task":"tg2-09","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:04:51Z","cost_usd":0.015951,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.26587079999999996,"task":"tg2-07","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:04:57Z","cost_usd":0.0201794,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.2860502,"task":"tg2-05","trial":1} +{"arm":"control","at":"2026-11-09T04:05:06Z","cost_usd":0.0289188,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.314969,"task":"tg2-01","trial":2} +{"arm":"control","at":"2026-11-09T04:05:16Z","cost_usd":0.0276803,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.3426493,"task":"tg2-04","trial":5} +{"arm":"control","at":"2026-11-09T04:05:27Z","cost_usd":0.0133582,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.3560075,"task":"tg2-07","trial":1} +{"arm":"control","at":"2026-11-09T04:05:33Z","cost_usd":0.0289504,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.38495789999999996,"task":"tg2-03","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:05:42Z","cost_usd":0.024409399999999998,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.4093673,"task":"tg2-09","trial":3} +{"arm":"control","at":"2026-11-09T04:05:49Z","cost_usd":0.0381462,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.4475135,"task":"tg2-02","trial":5} +{"arm":"control","at":"2026-11-09T04:06:01Z","cost_usd":0.0220941,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.4696076,"task":"tg2-09","trial":2} +{"arm":"control","at":"2026-11-09T04:06:08Z","cost_usd":0.0191077,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.4887153,"task":"tg2-10","trial":2} +{"arm":"control","at":"2026-11-09T04:06:15Z","cost_usd":0.024260900000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.5129762,"task":"tg2-04","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:06:24Z","cost_usd":0.030145799999999997,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.543122,"task":"tg2-02","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:06:32Z","cost_usd":0.024585300000000004,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.5677073,"task":"tg2-01","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:06:41Z","cost_usd":0.028692300000000004,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.5963996,"task":"tg2-09","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:06:49Z","cost_usd":0.0361465,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.6325461,"task":"tg2-02","trial":2} +{"arm":"control","at":"2026-11-09T04:07:00Z","cost_usd":0.029389300000000004,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.6619354,"task":"tg2-03","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:07:08Z","cost_usd":0.030078,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.6920134,"task":"tg2-03","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:07:15Z","cost_usd":0.024557799999999998,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.7165712,"task":"tg2-09","trial":5} +{"arm":"control","at":"2026-11-09T04:07:23Z","cost_usd":0.0246744,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.7412456,"task":"tg2-01","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:07:31Z","cost_usd":0.023905700000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.7651513,"task":"tg2-01","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:07:39Z","cost_usd":0.0276767,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.792828,"task":"tg2-01","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:07:47Z","cost_usd":0.0273452,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.8201731999999999,"task":"tg2-01","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:07:55Z","cost_usd":0.0159674,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.8361405999999999,"task":"tg2-07","trial":4} +{"arm":"control","at":"2026-11-09T04:08:00Z","cost_usd":0.0287332,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.8648737999999999,"task":"tg2-01","trial":3} +{"arm":"control","at":"2026-11-09T04:08:09Z","cost_usd":0.0134239,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.8782977,"task":"tg2-08","trial":5} +{"arm":"control","at":"2026-11-09T04:08:14Z","cost_usd":0.0190214,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.8973190999999999,"task":"tg2-06","trial":1} +{"arm":"control","at":"2026-11-09T04:08:19Z","cost_usd":0.0134422,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.9107612999999999,"task":"tg2-07","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:08:25Z","cost_usd":0.0156267,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.9263879999999999,"task":"tg2-08","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:08:31Z","cost_usd":0.0193826,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.9457705999999999,"task":"tg2-06","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:08:37Z","cost_usd":0.0201794,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.9659499999999999,"task":"tg2-05","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:08:45Z","cost_usd":0.026681700000000003,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":0.9926316999999999,"task":"tg2-01","trial":5} +{"arm":"control","at":"2026-11-09T04:08:55Z","cost_usd":0.0318161,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.0244478,"task":"tg2-02","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:09:04Z","cost_usd":0.0193826,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.0438303999999998,"task":"tg2-06","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:09:09Z","cost_usd":0.0329066,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.0767369999999998,"task":"tg2-03","trial":4} +{"arm":"control","at":"2026-11-09T04:09:18Z","cost_usd":0.0246817,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.1014186999999998,"task":"tg2-04","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:09:27Z","cost_usd":0.0156431,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.1170617999999997,"task":"tg2-08","trial":1} +{"arm":"control","at":"2026-11-09T04:09:32Z","cost_usd":0.0134939,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.1305556999999997,"task":"tg2-08","trial":2} +{"arm":"control","at":"2026-11-09T04:09:37Z","cost_usd":0.0190214,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.1495770999999997,"task":"tg2-06","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:09:42Z","cost_usd":0.0193826,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.1689596999999996,"task":"tg2-06","trial":4} +{"arm":"control","at":"2026-11-09T04:09:47Z","cost_usd":0.0161814,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.1851410999999996,"task":"tg2-10","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:09:51Z","cost_usd":0.0155531,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.2006941999999996,"task":"tg2-08","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:09:58Z","cost_usd":0.0159674,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.2166615999999997,"task":"tg2-07","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:10:03Z","cost_usd":0.020329800000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.2369913999999997,"task":"tg2-05","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:10:12Z","cost_usd":0.022963400000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.2599547999999998,"task":"tg2-04","trial":1} +{"arm":"control","at":"2026-11-09T04:10:21Z","cost_usd":0.0131357,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.2730905,"task":"tg2-08","trial":1} +{"arm":"control","at":"2026-11-09T04:10:27Z","cost_usd":0.025434699999999998,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.2985251999999998,"task":"tg2-09","trial":4} +{"arm":"control","at":"2026-11-09T04:10:36Z","cost_usd":0.0395019,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.3380271,"task":"tg2-02","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:10:43Z","cost_usd":0.032344,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.3703710999999998,"task":"tg2-03","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:10:51Z","cost_usd":0.0288152,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.3991862999999998,"task":"tg2-03","trial":1} +{"arm":"control","at":"2026-11-09T04:11:00Z","cost_usd":0.0133582,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.4125444999999999,"task":"tg2-07","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:11:05Z","cost_usd":0.0200646,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.4326090999999999,"task":"tg2-06","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:11:11Z","cost_usd":0.021625699999999998,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.4542347999999998,"task":"tg2-04","trial":4} +{"arm":"control","at":"2026-11-09T04:11:19Z","cost_usd":0.0190252,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.4732599999999998,"task":"tg2-06","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:11:24Z","cost_usd":0.0225125,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.4957724999999997,"task":"tg2-10","trial":2} +{"arm":"control","at":"2026-11-09T04:11:30Z","cost_usd":0.026664999999999998,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.5224374999999997,"task":"tg2-09","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:11:39Z","cost_usd":0.024001500000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.5464389999999997,"task":"tg2-04","trial":2} +{"arm":"control","at":"2026-11-09T04:11:50Z","cost_usd":0.015535,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.5619739999999998,"task":"tg2-05","trial":2} +{"arm":"control","at":"2026-11-09T04:11:55Z","cost_usd":0.021097099999999997,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.5830710999999997,"task":"tg2-10","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:12:05Z","cost_usd":0.020318299999999997,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.6033893999999997,"task":"tg2-05","trial":2} +{"arm":"control","at":"2026-11-09T04:12:12Z","cost_usd":0.0314825,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.6348718999999998,"task":"tg2-03","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:12:21Z","cost_usd":0.020511299999999996,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.6553831999999997,"task":"tg2-05","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:12:30Z","cost_usd":0.0217354,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.6771185999999998,"task":"tg2-10","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:12:36Z","cost_usd":0.034023,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.7111415999999997,"task":"tg2-02","trial":3} +{"arm":"control","at":"2026-11-09T04:12:45Z","cost_usd":0.0134157,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.7245572999999996,"task":"tg2-08","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:12:49Z","cost_usd":0.029080800000000004,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.7536380999999996,"task":"tg2-02","trial":4} +{"arm":"control","at":"2026-11-09T04:12:56Z","cost_usd":0.0133439,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.7669819999999996,"task":"tg2-08","trial":3} +{"arm":"control","at":"2026-11-09T04:13:00Z","cost_usd":0.0186751,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.7856570999999997,"task":"tg2-04","trial":4} +{"arm":"control","at":"2026-11-09T04:13:09Z","cost_usd":0.015563000000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.8012200999999997,"task":"tg2-05","trial":1} +{"arm":"control","at":"2026-11-09T04:13:14Z","cost_usd":0.0190214,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.8202414999999996,"task":"tg2-06","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:13:18Z","cost_usd":0.015623100000000001,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.8358645999999996,"task":"tg2-08","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:13:24Z","cost_usd":0.0286513,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.8645158999999996,"task":"tg2-03","trial":2} +{"arm":"control","at":"2026-11-09T04:13:32Z","cost_usd":0.018310800000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.8828266999999996,"task":"tg2-05","trial":3} +{"arm":"control","at":"2026-11-09T04:13:36Z","cost_usd":0.0212161,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.9040427999999996,"task":"tg2-09","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:13:43Z","cost_usd":0.034745200000000004,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.9387879999999995,"task":"tg2-02","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:13:55Z","cost_usd":0.0237615,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.9625494999999995,"task":"tg2-04","trial":5} +{"arm":"control","at":"2026-11-09T04:14:03Z","cost_usd":0.0358053,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":1.9983547999999995,"task":"tg2-02","trial":3} +{"arm":"control","at":"2026-11-09T04:14:12Z","cost_usd":0.027750500000000004,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.0261052999999993,"task":"tg2-03","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:14:21Z","cost_usd":0.0159674,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.0420726999999994,"task":"tg2-07","trial":1} +{"arm":"control","at":"2026-11-09T04:14:26Z","cost_usd":0.024787599999999996,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.066860299999999,"task":"tg2-04","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:14:34Z","cost_usd":0.020326200000000003,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.087186499999999,"task":"tg2-04","trial":3} +{"arm":"control","at":"2026-11-09T04:14:40Z","cost_usd":0.023942099999999997,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.1111285999999994,"task":"tg2-09","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:14:47Z","cost_usd":0.0193826,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.1305111999999995,"task":"tg2-06","trial":2} +{"arm":"control","at":"2026-11-09T04:14:52Z","cost_usd":0.0257164,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.1562275999999994,"task":"tg2-03","trial":3} +{"arm":"control","at":"2026-11-09T04:14:59Z","cost_usd":0.015514799999999999,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.1717423999999994,"task":"tg2-05","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:15:05Z","cost_usd":0.024738999999999997,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.1964813999999993,"task":"tg2-09","trial":4} +{"arm":"control","at":"2026-11-09T04:15:13Z","cost_usd":0.0200154,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.2164967999999994,"task":"tg2-06","trial":5} +{"arm":"control","at":"2026-11-09T04:15:17Z","cost_usd":0.028745200000000002,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.2452419999999993,"task":"tg2-01","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:15:26Z","cost_usd":0.0195013,"model":"claude-sonnet-5-5","repo":"theauth-go","running_total_usd":2.2647432999999992,"task":"tg2-10","trial":5} +{"arm":"control","at":"2026-11-09T04:15:35Z","cost_usd":0.0174919,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.2822351999999992,"task":"fd2-07","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:15:40Z","cost_usd":0.0169694,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.299204599999999,"task":"fd2-08","trial":4} +{"arm":"control","at":"2026-11-09T04:15:45Z","cost_usd":0.0215878,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.320792399999999,"task":"fd2-02","trial":1} +{"arm":"control","at":"2026-11-09T04:15:52Z","cost_usd":0.0169581,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.337750499999999,"task":"fd2-10","trial":5} +{"arm":"control","at":"2026-11-09T04:15:57Z","cost_usd":0.0186821,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.356432599999999,"task":"fd2-10","trial":4} +{"arm":"control","at":"2026-11-09T04:16:03Z","cost_usd":0.0175479,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.3739804999999987,"task":"fd2-07","trial":2} +{"arm":"control","at":"2026-11-09T04:16:08Z","cost_usd":0.0320306,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.406011099999999,"task":"fd2-01","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:16:14Z","cost_usd":0.020633899999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.426644999999999,"task":"fd2-10","trial":1} +{"arm":"control","at":"2026-11-09T04:16:20Z","cost_usd":0.0138707,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.440515699999999,"task":"fd2-05","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:16:24Z","cost_usd":0.0203238,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.4608394999999987,"task":"fd2-07","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:16:31Z","cost_usd":0.019304699999999998,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.480144199999999,"task":"fd2-10","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:16:37Z","cost_usd":0.0208878,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.501031999999999,"task":"fd2-09","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:16:44Z","cost_usd":0.019741799999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.520773799999999,"task":"fd2-07","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:16:51Z","cost_usd":0.0159926,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.536766399999999,"task":"fd2-05","trial":1} +{"arm":"control","at":"2026-11-09T04:16:56Z","cost_usd":0.0255315,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.562297899999999,"task":"fd2-01","trial":2} +{"arm":"control","at":"2026-11-09T04:17:01Z","cost_usd":0.021506699999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.5838045999999992,"task":"fd2-04","trial":5} +{"arm":"control","at":"2026-11-09T04:17:07Z","cost_usd":0.0179279,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.6017324999999993,"task":"fd2-07","trial":1} +{"arm":"control","at":"2026-11-09T04:17:12Z","cost_usd":0.0202609,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.621993399999999,"task":"fd2-03","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:17:20Z","cost_usd":0.017219699999999998,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.639213099999999,"task":"fd2-09","trial":3} +{"arm":"control","at":"2026-11-09T04:17:28Z","cost_usd":0.021678400000000004,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.660891499999999,"task":"fd2-02","trial":5} +{"arm":"control","at":"2026-11-09T04:17:34Z","cost_usd":0.0186761,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.679567599999999,"task":"fd2-09","trial":2} +{"arm":"control","at":"2026-11-09T04:17:39Z","cost_usd":0.0169581,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.696525699999999,"task":"fd2-10","trial":2} +{"arm":"control","at":"2026-11-09T04:17:45Z","cost_usd":0.0191511,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.7156767999999993,"task":"fd2-04","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:17:51Z","cost_usd":0.0233581,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.739034899999999,"task":"fd2-02","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:18:00Z","cost_usd":0.029403199999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.768438099999999,"task":"fd2-01","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:18:07Z","cost_usd":0.0212633,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.7897013999999993,"task":"fd2-09","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:18:14Z","cost_usd":0.020276,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.8099773999999993,"task":"fd2-02","trial":2} +{"arm":"control","at":"2026-11-09T04:18:21Z","cost_usd":0.0196685,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.829645899999999,"task":"fd2-03","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:18:27Z","cost_usd":0.0226275,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.852273399999999,"task":"fd2-03","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:18:35Z","cost_usd":0.0215449,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.873818299999999,"task":"fd2-09","trial":5} +{"arm":"control","at":"2026-11-09T04:18:43Z","cost_usd":0.0232428,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.897061099999999,"task":"fd2-01","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:18:50Z","cost_usd":0.019877600000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.916938699999999,"task":"fd2-01","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:18:56Z","cost_usd":0.019877600000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.936816299999999,"task":"fd2-01","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:19:01Z","cost_usd":0.021228200000000003,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.958044499999999,"task":"fd2-01","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:19:09Z","cost_usd":0.0187198,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":2.976764299999999,"task":"fd2-07","trial":4} +{"arm":"control","at":"2026-11-09T04:19:14Z","cost_usd":0.030179599999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.0069438999999987,"task":"fd2-01","trial":3} +{"arm":"control","at":"2026-11-09T04:19:21Z","cost_usd":0.0154397,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.0223835999999986,"task":"fd2-08","trial":5} +{"arm":"control","at":"2026-11-09T04:19:26Z","cost_usd":0.0121416,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.0345251999999987,"task":"fd2-06","trial":1} +{"arm":"control","at":"2026-11-09T04:19:29Z","cost_usd":0.0179597,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.0524848999999987,"task":"fd2-07","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:19:34Z","cost_usd":0.0178514,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.070336299999999,"task":"fd2-08","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:19:41Z","cost_usd":0.0165909,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.086927199999999,"task":"fd2-06","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:19:47Z","cost_usd":0.0159762,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.102903399999999,"task":"fd2-05","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:19:52Z","cost_usd":0.018760500000000003,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.121663899999999,"task":"fd2-01","trial":5} +{"arm":"control","at":"2026-11-09T04:19:59Z","cost_usd":0.0215878,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.1432516999999986,"task":"fd2-02","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:20:05Z","cost_usd":0.014791499999999999,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.1580431999999985,"task":"fd2-06","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:20:10Z","cost_usd":0.022916699999999998,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.1809598999999986,"task":"fd2-03","trial":4} +{"arm":"control","at":"2026-11-09T04:20:19Z","cost_usd":0.0224964,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.2034562999999987,"task":"fd2-04","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:20:24Z","cost_usd":0.0210284,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.2244846999999988,"task":"fd2-08","trial":1} +{"arm":"control","at":"2026-11-09T04:20:31Z","cost_usd":0.0164479,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.240932599999999,"task":"fd2-08","trial":2} +{"arm":"control","at":"2026-11-09T04:20:37Z","cost_usd":0.0121256,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.253058199999999,"task":"fd2-06","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:20:40Z","cost_usd":0.0147815,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.267839699999999,"task":"fd2-06","trial":4} +{"arm":"control","at":"2026-11-09T04:20:47Z","cost_usd":0.0169581,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.284797799999999,"task":"fd2-10","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:20:52Z","cost_usd":0.0175458,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.302343599999999,"task":"fd2-08","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:20:58Z","cost_usd":0.0195158,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.3218593999999992,"task":"fd2-07","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:21:04Z","cost_usd":0.0159926,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.3378519999999994,"task":"fd2-05","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:21:10Z","cost_usd":0.0221282,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.3599801999999994,"task":"fd2-04","trial":1} +{"arm":"control","at":"2026-11-09T04:21:17Z","cost_usd":0.015425900000000001,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.3754060999999993,"task":"fd2-08","trial":1} +{"arm":"control","at":"2026-11-09T04:21:21Z","cost_usd":0.019726,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.395132099999999,"task":"fd2-09","trial":4} +{"arm":"control","at":"2026-11-09T04:21:28Z","cost_usd":0.0212969,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.416428999999999,"task":"fd2-02","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:21:35Z","cost_usd":0.023411500000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.439840499999999,"task":"fd2-03","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:21:42Z","cost_usd":0.0227353,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.462575799999999,"task":"fd2-03","trial":1} +{"arm":"control","at":"2026-11-09T04:21:50Z","cost_usd":0.0174919,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.480067699999999,"task":"fd2-07","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:21:55Z","cost_usd":0.014751499999999999,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.494819199999999,"task":"fd2-06","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:22:01Z","cost_usd":0.018660200000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.5134793999999987,"task":"fd2-04","trial":4} +{"arm":"control","at":"2026-11-09T04:22:08Z","cost_usd":0.012585600000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.5260649999999987,"task":"fd2-06","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:22:12Z","cost_usd":0.0206507,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.5467156999999987,"task":"fd2-10","trial":2} +{"arm":"control","at":"2026-11-09T04:22:20Z","cost_usd":0.01833,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.565045699999999,"task":"fd2-09","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:22:25Z","cost_usd":0.0230354,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.588081099999999,"task":"fd2-04","trial":2} +{"arm":"control","at":"2026-11-09T04:22:34Z","cost_usd":0.0138707,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.601951799999999,"task":"fd2-05","trial":2} +{"arm":"control","at":"2026-11-09T04:22:38Z","cost_usd":0.0182574,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.620209199999999,"task":"fd2-10","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:22:43Z","cost_usd":0.0159926,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.636201799999999,"task":"fd2-05","trial":2} +{"arm":"control","at":"2026-11-09T04:22:49Z","cost_usd":0.0194889,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.6556906999999987,"task":"fd2-03","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:22:55Z","cost_usd":0.0159926,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.671683299999999,"task":"fd2-05","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:23:00Z","cost_usd":0.0206367,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.6923199999999987,"task":"fd2-10","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:23:06Z","cost_usd":0.023392,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.7157119999999986,"task":"fd2-02","trial":3} +{"arm":"control","at":"2026-11-09T04:23:15Z","cost_usd":0.0155199,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.7312318999999987,"task":"fd2-08","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:23:19Z","cost_usd":0.0233545,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.7545863999999987,"task":"fd2-02","trial":4} +{"arm":"control","at":"2026-11-09T04:23:28Z","cost_usd":0.015395900000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.769982299999999,"task":"fd2-08","trial":3} +{"arm":"control","at":"2026-11-09T04:23:32Z","cost_usd":0.0179853,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.7879675999999987,"task":"fd2-04","trial":4} +{"arm":"control","at":"2026-11-09T04:23:37Z","cost_usd":0.0138707,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.8018382999999987,"task":"fd2-05","trial":1} +{"arm":"control","at":"2026-11-09T04:23:41Z","cost_usd":0.012523399999999999,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.8143616999999987,"task":"fd2-06","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:23:45Z","cost_usd":0.017001799999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.831363499999999,"task":"fd2-08","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:23:51Z","cost_usd":0.022494,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.853857499999999,"task":"fd2-03","trial":2} +{"arm":"control","at":"2026-11-09T04:24:00Z","cost_usd":0.0138867,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.867744199999999,"task":"fd2-05","trial":3} +{"arm":"control","at":"2026-11-09T04:24:04Z","cost_usd":0.017890200000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.885634399999999,"task":"fd2-09","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:24:10Z","cost_usd":0.0233545,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.908988899999999,"task":"fd2-02","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:24:18Z","cost_usd":0.018660200000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.9276490999999987,"task":"fd2-04","trial":5} +{"arm":"control","at":"2026-11-09T04:24:25Z","cost_usd":0.021378099999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.949027199999999,"task":"fd2-02","trial":3} +{"arm":"control","at":"2026-11-09T04:24:38Z","cost_usd":0.0205629,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.9695900999999987,"task":"fd2-03","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:24:43Z","cost_usd":0.0198354,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":3.9894254999999985,"task":"fd2-07","trial":1} +{"arm":"control","at":"2026-11-09T04:24:51Z","cost_usd":0.0179853,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.007410799999999,"task":"fd2-04","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:24:55Z","cost_usd":0.018660200000000002,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.026070999999999,"task":"fd2-04","trial":3} +{"arm":"control","at":"2026-11-09T04:25:01Z","cost_usd":0.0180664,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.044137399999999,"task":"fd2-09","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:25:08Z","cost_usd":0.014801499999999999,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.058938899999999,"task":"fd2-06","trial":2} +{"arm":"control","at":"2026-11-09T04:25:14Z","cost_usd":0.020224899999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.079163799999999,"task":"fd2-03","trial":3} +{"arm":"control","at":"2026-11-09T04:25:21Z","cost_usd":0.0138707,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.093034499999999,"task":"fd2-05","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:25:25Z","cost_usd":0.0175173,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.110551799999999,"task":"fd2-09","trial":4} +{"arm":"control","at":"2026-11-09T04:25:30Z","cost_usd":0.0127116,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.123263399999999,"task":"fd2-06","trial":5} +{"arm":"control","at":"2026-11-09T04:25:34Z","cost_usd":0.020183399999999997,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.143446799999999,"task":"fd2-01","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:25:40Z","cost_usd":0.019304699999999998,"model":"claude-sonnet-5-5","repo":"featuredrop","running_total_usd":4.162751499999999,"task":"fd2-10","trial":5} +{"arm":"control","at":"2026-11-09T04:25:48Z","cost_usd":0.015937800000000002,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.1786892999999985,"task":"ta-07","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:25:53Z","cost_usd":0.0178856,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.196574899999998,"task":"ta-08","trial":4} +{"arm":"control","at":"2026-11-09T04:26:06Z","cost_usd":0.0157149,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.212289799999998,"task":"ta-02","trial":1} +{"arm":"control","at":"2026-11-09T04:26:13Z","cost_usd":0.018211,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.230500799999998,"task":"ta-10","trial":5} +{"arm":"control","at":"2026-11-09T04:26:20Z","cost_usd":0.0182006,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.248701399999998,"task":"ta-10","trial":4} +{"arm":"control","at":"2026-11-09T04:26:26Z","cost_usd":0.015958,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.264659399999998,"task":"ta-07","trial":2} +{"arm":"control","at":"2026-11-09T04:26:35Z","cost_usd":0.0167944,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.281453799999999,"task":"ta-01","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:26:42Z","cost_usd":0.0169669,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.2984206999999985,"task":"ta-10","trial":1} +{"arm":"control","at":"2026-11-09T04:26:55Z","cost_usd":0.0185138,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.316934499999999,"task":"ta-05","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:27:02Z","cost_usd":0.0186712,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.335605699999999,"task":"ta-07","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:27:14Z","cost_usd":0.0190519,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.354657599999999,"task":"ta-10","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:27:27Z","cost_usd":0.0248475,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.3795050999999985,"task":"ta-09","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:27:44Z","cost_usd":0.0186954,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.398200499999999,"task":"ta-07","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:28:05Z","cost_usd":0.019684,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.417884499999999,"task":"ta-05","trial":1} +{"arm":"control","at":"2026-11-09T04:28:25Z","cost_usd":0.014329999999999999,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.432214499999999,"task":"ta-01","trial":2} +{"arm":"control","at":"2026-11-09T04:28:30Z","cost_usd":0.019168400000000002,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.451382899999999,"task":"ta-04","trial":5} +{"arm":"control","at":"2026-11-09T04:28:35Z","cost_usd":0.015958,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.467340899999999,"task":"ta-07","trial":1} +{"arm":"control","at":"2026-11-09T04:28:40Z","cost_usd":0.0172502,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.484591099999999,"task":"ta-03","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:28:46Z","cost_usd":0.022717600000000004,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.507308699999999,"task":"ta-09","trial":3} +{"arm":"control","at":"2026-11-09T04:29:00Z","cost_usd":0.0151909,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.5224996,"task":"ta-02","trial":5} +{"arm":"control","at":"2026-11-09T04:29:06Z","cost_usd":0.020594599999999998,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.5430942,"task":"ta-09","trial":2} +{"arm":"control","at":"2026-11-09T04:29:12Z","cost_usd":0.018211,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.5613052,"task":"ta-10","trial":2} +{"arm":"control","at":"2026-11-09T04:29:18Z","cost_usd":0.0200042,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.581309399999999,"task":"ta-04","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:29:22Z","cost_usd":0.0152256,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.596534999999999,"task":"ta-02","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:29:34Z","cost_usd":0.0195138,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.6160488,"task":"ta-01","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:29:48Z","cost_usd":0.024628699999999996,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.6406775,"task":"ta-09","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:30:04Z","cost_usd":0.014962,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.6556394999999995,"task":"ta-02","trial":2} +{"arm":"control","at":"2026-11-09T04:30:16Z","cost_usd":0.0175673,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.673206799999999,"task":"ta-03","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:30:26Z","cost_usd":0.0196334,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.692840199999999,"task":"ta-03","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:30:40Z","cost_usd":0.022998900000000003,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.715839099999999,"task":"ta-09","trial":5} +{"arm":"control","at":"2026-11-09T04:30:53Z","cost_usd":0.016752799999999998,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.732591899999999,"task":"ta-01","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:31:00Z","cost_usd":0.0172756,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.749867499999999,"task":"ta-01","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:31:12Z","cost_usd":0.019662,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.769529499999999,"task":"ta-01","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:31:25Z","cost_usd":0.0189098,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.788439299999999,"task":"ta-01","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:31:38Z","cost_usd":0.021775700000000002,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.8102149999999995,"task":"ta-07","trial":4} +{"arm":"control","at":"2026-11-09T04:31:54Z","cost_usd":0.01456,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.824775,"task":"ta-01","trial":3} +{"arm":"control","at":"2026-11-09T04:32:00Z","cost_usd":0.015444399999999999,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.8402194,"task":"ta-08","trial":5} +{"arm":"control","at":"2026-11-09T04:32:04Z","cost_usd":0.013002300000000001,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.8532217,"task":"ta-06","trial":1} +{"arm":"control","at":"2026-11-09T04:32:08Z","cost_usd":0.015958,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.8691797,"task":"ta-07","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:32:13Z","cost_usd":0.0179436,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.8871233,"task":"ta-08","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:32:25Z","cost_usd":0.0152079,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.9023312,"task":"ta-06","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:32:37Z","cost_usd":0.020412,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.9227432,"task":"ta-05","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:32:52Z","cost_usd":0.0161672,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.9389104,"task":"ta-01","trial":5} +{"arm":"control","at":"2026-11-09T04:33:04Z","cost_usd":0.0128404,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.9517508,"task":"ta-02","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:33:10Z","cost_usd":0.0152079,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.9669587,"task":"ta-06","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:33:23Z","cost_usd":0.020039400000000002,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":4.9869981,"task":"ta-03","trial":4} +{"arm":"control","at":"2026-11-09T04:33:36Z","cost_usd":0.0175004,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.0044985,"task":"ta-04","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:33:42Z","cost_usd":0.01795,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.0224485,"task":"ta-08","trial":1} +{"arm":"control","at":"2026-11-09T04:33:56Z","cost_usd":0.015444399999999999,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.0378929,"task":"ta-08","trial":2} +{"arm":"control","at":"2026-11-09T04:34:01Z","cost_usd":0.013002300000000001,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.0508952,"task":"ta-06","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:34:06Z","cost_usd":0.015215699999999999,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.0661109,"task":"ta-06","trial":4} +{"arm":"control","at":"2026-11-09T04:34:24Z","cost_usd":0.020237,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.0863479,"task":"ta-10","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:34:30Z","cost_usd":0.017944,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.1042919,"task":"ta-08","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:34:43Z","cost_usd":0.021224500000000004,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.1255163999999995,"task":"ta-07","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:34:59Z","cost_usd":0.01967,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.145186399999999,"task":"ta-05","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:35:14Z","cost_usd":0.018163600000000002,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.163349999999999,"task":"ta-04","trial":1} +{"arm":"control","at":"2026-11-09T04:35:28Z","cost_usd":0.0153884,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.178738399999999,"task":"ta-08","trial":1} +{"arm":"control","at":"2026-11-09T04:35:33Z","cost_usd":0.0206046,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.199343,"task":"ta-09","trial":4} +{"arm":"control","at":"2026-11-09T04:35:39Z","cost_usd":0.0128404,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.2121834,"task":"ta-02","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:35:44Z","cost_usd":0.0283196,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.2405029999999995,"task":"ta-03","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:35:59Z","cost_usd":0.0201999,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.260702899999999,"task":"ta-03","trial":1} +{"arm":"control","at":"2026-11-09T04:36:14Z","cost_usd":0.016248,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.276950899999999,"task":"ta-07","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:36:19Z","cost_usd":0.0152079,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.292158799999999,"task":"ta-06","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:36:31Z","cost_usd":0.017906000000000002,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.310064799999999,"task":"ta-04","trial":4} +{"arm":"control","at":"2026-11-09T04:36:43Z","cost_usd":0.013002300000000001,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.323067099999999,"task":"ta-06","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:36:47Z","cost_usd":0.020989,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.3440560999999995,"task":"ta-10","trial":2} +{"arm":"control","at":"2026-11-09T04:37:01Z","cost_usd":0.0192402,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.363296299999999,"task":"ta-09","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:37:07Z","cost_usd":0.018164,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.381460299999999,"task":"ta-04","trial":2} +{"arm":"control","at":"2026-11-09T04:37:19Z","cost_usd":0.0231114,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.404571699999999,"task":"ta-05","trial":2} +{"arm":"control","at":"2026-11-09T04:37:25Z","cost_usd":0.0198748,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.424446499999999,"task":"ta-10","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:37:32Z","cost_usd":0.0204941,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.444940599999999,"task":"ta-05","trial":2} +{"arm":"control","at":"2026-11-09T04:37:45Z","cost_usd":0.0171169,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.462057499999999,"task":"ta-03","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:37:51Z","cost_usd":0.020409999999999998,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.482467499999999,"task":"ta-05","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:38:05Z","cost_usd":0.0226928,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.505160299999999,"task":"ta-10","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:38:19Z","cost_usd":0.014836,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.519996299999999,"task":"ta-02","trial":3} +{"arm":"control","at":"2026-11-09T04:38:31Z","cost_usd":0.015444399999999999,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.535440699999999,"task":"ta-08","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:38:35Z","cost_usd":0.0148056,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.550246299999999,"task":"ta-02","trial":4} +{"arm":"control","at":"2026-11-09T04:38:51Z","cost_usd":0.0153884,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.565634699999999,"task":"ta-08","trial":3} +{"arm":"control","at":"2026-11-09T04:38:56Z","cost_usd":0.0192202,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.584854899999999,"task":"ta-04","trial":4} +{"arm":"control","at":"2026-11-09T04:39:01Z","cost_usd":0.016319100000000003,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.6011739999999985,"task":"ta-05","trial":1} +{"arm":"control","at":"2026-11-09T04:39:07Z","cost_usd":0.013002300000000001,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.614176299999999,"task":"ta-06","trial":3} +{"arm":"stacklit-quiet","at":"2026-11-09T04:39:11Z","cost_usd":0.01795,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.632126299999999,"task":"ta-08","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:39:22Z","cost_usd":0.0194919,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.651618199999999,"task":"ta-03","trial":2} +{"arm":"control","at":"2026-11-09T04:39:37Z","cost_usd":0.0172931,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.668911299999999,"task":"ta-05","trial":3} +{"arm":"control","at":"2026-11-09T04:39:42Z","cost_usd":0.0206346,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.689545899999999,"task":"ta-09","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:39:50Z","cost_usd":0.0185022,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.708048099999999,"task":"ta-02","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:40:02Z","cost_usd":0.018164,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.726212099999999,"task":"ta-04","trial":5} +{"arm":"control","at":"2026-11-09T04:40:14Z","cost_usd":0.0128322,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.739044299999999,"task":"ta-02","trial":3} +{"arm":"control","at":"2026-11-09T04:40:18Z","cost_usd":0.017483300000000004,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.756527599999999,"task":"ta-03","trial":2} +{"arm":"stacklit-quiet","at":"2026-11-09T04:40:25Z","cost_usd":0.0185832,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.775110799999999,"task":"ta-07","trial":1} +{"arm":"control","at":"2026-11-09T04:40:37Z","cost_usd":0.018134400000000002,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.793245199999999,"task":"ta-04","trial":1} +{"arm":"stacklit-quiet","at":"2026-11-09T04:40:42Z","cost_usd":0.018164,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.811409199999999,"task":"ta-04","trial":3} +{"arm":"control","at":"2026-11-09T04:40:54Z","cost_usd":0.0204184,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.831827599999999,"task":"ta-09","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:41:03Z","cost_usd":0.0152079,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.847035499999999,"task":"ta-06","trial":2} +{"arm":"control","at":"2026-11-09T04:41:15Z","cost_usd":0.0170502,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.864085699999999,"task":"ta-03","trial":3} +{"arm":"control","at":"2026-11-09T04:41:22Z","cost_usd":0.0166391,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.8807247999999985,"task":"ta-05","trial":5} +{"arm":"stacklit-quiet","at":"2026-11-09T04:41:29Z","cost_usd":0.0229589,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.903683699999998,"task":"ta-09","trial":4} +{"arm":"control","at":"2026-11-09T04:41:44Z","cost_usd":0.013002300000000001,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.916685999999999,"task":"ta-06","trial":5} +{"arm":"control","at":"2026-11-09T04:41:49Z","cost_usd":0.014315999999999999,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.9310019999999986,"task":"ta-01","trial":4} +{"arm":"stacklit-quiet","at":"2026-11-09T04:41:54Z","cost_usd":0.0182363,"model":"claude-sonnet-5-5","repo":"theauth","running_total_usd":5.949238299999998,"task":"ta-10","trial":5} diff --git a/bench/agent-ab/round3/summary.json b/bench/agent-ab/round3/summary.json new file mode 100644 index 0000000..2001643 --- /dev/null +++ b/bench/agent-ab/round3/summary.json @@ -0,0 +1,2743 @@ +{ + "date": "2026-11-09", + "model": "claude-sonnet-5-5", + "bootstrap_seed": 20260508, + "bootstrap_resamples": 10000, + "ledger_total_notional_usd": 5.949238299999998, + "sessions_per_arm": { + "control": 150, + "memory": 0, + "stacklit": 0, + "stacklit-e": 0, + "stacklit-e-small": 0, + "stacklit-quiet": 150, + "strict": 0 + }, + "memorisation_check_correct": { + "featuredrop": { + "k": 0, + "n": 0, + "rate": 0, + "wilson95": { + "lo": 0, + "hi": 0 + } + }, + "levelrail": { + "k": 0, + "n": 0, + "rate": 0, + "wilson95": { + "lo": 0, + "hi": 0 + } + }, + "theauth": { + "k": 0, + "n": 0, + "rate": 0, + "wilson95": { + "lo": 0, + "hi": 0 + } + }, + "theauth-go": { + "k": 0, + "n": 0, + "rate": 0, + "wilson95": { + "lo": 0, + "hi": 0 + } + } + }, + "scopes": [ + { + "name": "theauth-go", + "arm": "stacklit-quiet", + "base": "control", + "n_pairs": 50, + "comparisons": [ + { + "metric": "cost_usd", + "n_pairs": 50, + "mean_control": 0.022172563999999992, + "mean_arm": 0.023122301999999994, + "mean_diff": 0.000949738000000002, + "diff_ci95": { + "lo": -0.00006605800000000439, + "hi": 0.0019372699999999998 + }, + "ratio": 1.0428339275511844, + "ratio_ci95": { + "lo": 0.9971963412383539, + "hi": 1.0922746905400653 + } + }, + { + "metric": "total_tokens", + "n_pairs": 50, + "mean_control": 19115.86, + "mean_arm": 24672.64, + "mean_diff": 5556.779999999999, + "diff_ci95": { + "lo": 3536.78, + "hi": 7652.18 + }, + "ratio": 1.2906895112226182, + "ratio_ci95": { + "lo": 1.1772321680691251, + "hi": 1.4202899656370123 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 50, + "mean_control": 6036.304, + "mean_arm": 6699.081999999999, + "mean_diff": 662.7779999999993, + "diff_ci95": { + "lo": 343.07800000000395, + "hi": 979.0140000000002 + }, + "ratio": 1.1097986449986614, + "ratio_ci95": { + "lo": 1.0537908153841415, + "hi": 1.1709291436521183 + } + }, + { + "metric": "turns", + "n_pairs": 50, + "mean_control": 3.44, + "mean_arm": 3.44, + "mean_diff": 0, + "diff_ci95": { + "lo": -0.34, + "hi": 0.34 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 0.9067357512953368, + "hi": 1.1049382716049383 + } + }, + { + "metric": "tool_calls", + "n_pairs": 50, + "mean_control": 2.44, + "mean_arm": 2.44, + "mean_diff": 0, + "diff_ci95": { + "lo": -0.34, + "hi": 0.34 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 0.873015873015873, + "hi": 1.1565217391304348 + } + }, + { + "metric": "read_calls", + "n_pairs": 50, + "mean_control": 0.06, + "mean_arm": 0.02, + "mean_diff": -0.039999999999999994, + "diff_ci95": { + "lo": -0.12, + "hi": 0.04 + }, + "ratio": 0.33333333333333337, + "ratio_ci95": { + "lo": 0, + "hi": 2 + } + }, + { + "metric": "grep_glob_calls", + "n_pairs": 50, + "mean_control": 2.38, + "mean_arm": 2.36, + "mean_diff": -0.020000000000000018, + "diff_ci95": { + "lo": -0.38, + "hi": 0.34 + }, + "ratio": 0.9915966386554622, + "ratio_ci95": { + "lo": 0.8552631578947368, + "hi": 1.1538461538461537 + } + }, + { + "metric": "files_read", + "n_pairs": 50, + "mean_control": 0.06, + "mean_arm": 0.02, + "mean_diff": -0.039999999999999994, + "diff_ci95": { + "lo": -0.12, + "hi": 0.04 + }, + "ratio": 0.33333333333333337, + "ratio_ci95": { + "lo": 0, + "hi": 2 + } + }, + { + "metric": "read_kb", + "n_pairs": 50, + "mean_control": 0.3527734375, + "mean_arm": 0.01361328125, + "mean_diff": -0.33916015625, + "diff_ci95": { + "lo": -0.7978515625, + "hi": 0.01361328125 + }, + "ratio": 0.03858930351013177, + "ratio_ci95": { + "lo": 0, + "hi": 0.2663863940378368 + } + }, + { + "metric": "seconds", + "n_pairs": 50, + "mean_control": 7.010185800999998, + "mean_arm": 7.39300078912, + "mean_diff": 0.3828149881200025, + "diff_ci95": { + "lo": -0.21333965171999922, + "hi": 0.9800147148799999 + }, + "ratio": 1.0546083939837079, + "ratio_ci95": { + "lo": 0.9715803926599601, + "hi": 1.148215904809864 + } + }, + { + "metric": "correct", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "recall", + "n_pairs": 30, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "correctness": { + "control": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + } + }, + "correctness_audited": { + "control": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + } + }, + "stacklit_used_in_treatment": { + "k": 0, + "n": 50, + "rate": 0, + "wilson95": { + "lo": 0, + "hi": 0.07135003417431873 + } + }, + "hook_fired_in_treatment": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "by_kind": { + "add": [ + { + "metric": "cost_usd", + "n_pairs": 10, + "mean_control": 0.02132062, + "mean_arm": 0.023941289999999997, + "mean_diff": 0.0026206699999999986, + "diff_ci95": { + "lo": 0.0002362199999999981, + "hi": 0.005126800000000001 + }, + "ratio": 1.122917157193365, + "ratio_ci95": { + "lo": 1.0105104383576273, + "hi": 1.2544961335025504 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 10, + "mean_control": 5880.219999999999, + "mean_arm": 6920.589999999999, + "mean_diff": 1040.37, + "diff_ci95": { + "lo": 278.38000000000176, + "hi": 1821.9200000000005 + }, + "ratio": 1.1769270537496896, + "ratio_ci95": { + "lo": 1.044643025203757, + "hi": 1.3332485291213516 + } + }, + { + "metric": "tool_calls", + "n_pairs": 10, + "mean_control": 2.7, + "mean_arm": 2.8, + "mean_diff": 0.09999999999999964, + "diff_ci95": { + "lo": -0.7, + "hi": 1 + }, + "ratio": 1.037037037037037, + "ratio_ci95": { + "lo": 0.7741935483870968, + "hi": 1.4545454545454546 + } + }, + { + "metric": "files_read", + "n_pairs": 10, + "mean_control": 0, + "mean_arm": 0.1, + "mean_diff": 0.1, + "diff_ci95": { + "lo": 0, + "hi": 0.3 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "impact": [ + { + "metric": "cost_usd", + "n_pairs": 10, + "mean_control": 0.013386190000000001, + "mean_arm": 0.015777969999999995, + "mean_diff": 0.0023917799999999944, + "diff_ci95": { + "lo": 0.0022636600000000007, + "hi": 0.002508579999999999 + }, + "ratio": 1.1786751868903693, + "ratio_ci95": { + "lo": 1.1688715372384502, + "hi": 1.1876281168560288 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 10, + "mean_control": 3697.290000000001, + "mean_arm": 4486.469999999999, + "mean_diff": 789.1799999999985, + "diff_ci95": { + "lo": 772.0599999999991, + "hi": 805.2800000000003 + }, + "ratio": 1.2134482282969414, + "ratio_ci95": { + "lo": 1.2091959504866783, + "hi": 1.2174260332674895 + } + }, + { + "metric": "tool_calls", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "files_read", + "n_pairs": 10, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "locate": [ + { + "metric": "cost_usd", + "n_pairs": 20, + "mean_control": 0.029249834999999995, + "mean_arm": 0.027990465, + "mean_diff": -0.0012593699999999958, + "diff_ci95": { + "lo": -0.002980179999999999, + "hi": 0.00048547999999999923 + }, + "ratio": 0.9569443725067168, + "ratio_ci95": { + "lo": 0.9023276427175397, + "hi": 1.0168825667921808 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 20, + "mean_control": 7896.734999999999, + "mean_arm": 8025.265000000002, + "mean_diff": 128.53000000000338, + "diff_ci95": { + "lo": -388.8600000000035, + "hi": 635.4400000000009 + }, + "ratio": 1.0162763471232101, + "ratio_ci95": { + "lo": 0.9525390455844569, + "hi": 1.081912050636611 + } + }, + { + "metric": "tool_calls", + "n_pairs": 20, + "mean_control": 3.55, + "mean_arm": 3.2, + "mean_diff": -0.34999999999999964, + "diff_ci95": { + "lo": -1, + "hi": 0.3 + }, + "ratio": 0.9014084507042255, + "ratio_ci95": { + "lo": 0.7441860465116279, + "hi": 1.098360655737705 + } + }, + { + "metric": "files_read", + "n_pairs": 20, + "mean_control": 0.15, + "mean_arm": 0, + "mean_diff": -0.15, + "diff_ci95": { + "lo": -0.3, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 20, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 20, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "trace": [ + { + "metric": "cost_usd", + "n_pairs": 10, + "mean_control": 0.017656339999999996, + "mean_arm": 0.01991132, + "mean_diff": 0.0022549800000000036, + "diff_ci95": { + "lo": 0.0009553900000000032, + "hi": 0.003586860000000003 + }, + "ratio": 1.1277150304083408, + "ratio_ci95": { + "lo": 1.0509219817422406, + "hi": 1.2166049043166096 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 10, + "mean_control": 4810.54, + "mean_arm": 6037.82, + "mean_diff": 1227.2799999999997, + "diff_ci95": { + "lo": 628.0299999999995, + "hi": 1859.2400000000002 + }, + "ratio": 1.2551231254703215, + "ratio_ci95": { + "lo": 1.1231257547954012, + "hi": 1.4158366846098043 + } + }, + { + "metric": "tool_calls", + "n_pairs": 10, + "mean_control": 1.4, + "mean_arm": 2, + "mean_diff": 0.6000000000000001, + "diff_ci95": { + "lo": 0.2, + "hi": 1 + }, + "ratio": 1.4285714285714286, + "ratio_ci95": { + "lo": 1.1666666666666667, + "hi": 1.7142857142857142 + } + }, + { + "metric": "files_read", + "n_pairs": 10, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ] + } + }, + { + "name": "featuredrop", + "arm": "stacklit-quiet", + "base": "control", + "n_pairs": 50, + "comparisons": [ + { + "metric": "cost_usd", + "n_pairs": 50, + "mean_control": 0.01833272399999999, + "mean_arm": 0.019627439999999993, + "mean_diff": 0.0012947160000000013, + "diff_ci95": { + "lo": 0.00047227800000000376, + "hi": 0.0020126939999999993 + }, + "ratio": 1.0706232199862935, + "ratio_ci95": { + "lo": 1.0245850494884388, + "hi": 1.1147475045758992 + } + }, + { + "metric": "total_tokens", + "n_pairs": 50, + "mean_control": 16795.34, + "mean_arm": 20495.14, + "mean_diff": 3699.7999999999993, + "diff_ci95": { + "lo": 2617.22, + "hi": 4739.26 + }, + "ratio": 1.2202872939755909, + "ratio_ci95": { + "lo": 1.150805801386036, + "hi": 1.2946497652248692 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 50, + "mean_control": 5080.184000000001, + "mean_arm": 5689.059999999998, + "mean_diff": 608.8759999999966, + "diff_ci95": { + "lo": 378.56400000000025, + "hi": 804.9920000000006 + }, + "ratio": 1.1198531391776354, + "ratio_ci95": { + "lo": 1.0712906024463955, + "hi": 1.1647039437962594 + } + }, + { + "metric": "turns", + "n_pairs": 50, + "mean_control": 3.2, + "mean_arm": 2.96, + "mean_diff": -0.2400000000000002, + "diff_ci95": { + "lo": -0.42, + "hi": -0.06 + }, + "ratio": 0.9249999999999999, + "ratio_ci95": { + "lo": 0.8695652173913043, + "hi": 0.9806451612903225 + } + }, + { + "metric": "tool_calls", + "n_pairs": 50, + "mean_control": 2.2, + "mean_arm": 1.96, + "mean_diff": -0.2400000000000002, + "diff_ci95": { + "lo": -0.44, + "hi": -0.06 + }, + "ratio": 0.8909090909090909, + "ratio_ci95": { + "lo": 0.811965811965812, + "hi": 0.9716981132075472 + } + }, + { + "metric": "read_calls", + "n_pairs": 50, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "grep_glob_calls", + "n_pairs": 50, + "mean_control": 2.2, + "mean_arm": 1.9, + "mean_diff": -0.30000000000000027, + "diff_ci95": { + "lo": -0.5, + "hi": -0.12 + }, + "ratio": 0.8636363636363635, + "ratio_ci95": { + "lo": 0.7857142857142857, + "hi": 0.9439252336448598 + } + }, + { + "metric": "files_read", + "n_pairs": 50, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "read_kb", + "n_pairs": 50, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "seconds", + "n_pairs": 50, + "mean_control": 5.38784931172, + "mean_arm": 6.796949240820001, + "mean_diff": 1.4090999291000008, + "diff_ci95": { + "lo": 1.0216280465400007, + "hi": 1.7598895016200005 + }, + "ratio": 1.2615329137054436, + "ratio_ci95": { + "lo": 1.1782321809457148, + "hi": 1.344295223046229 + } + }, + { + "metric": "correct", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "recall", + "n_pairs": 35, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "correctness": { + "control": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + } + }, + "correctness_audited": { + "control": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + } + }, + "stacklit_used_in_treatment": { + "k": 0, + "n": 50, + "rate": 0, + "wilson95": { + "lo": 0, + "hi": 0.07135003417431873 + } + }, + "hook_fired_in_treatment": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "by_kind": { + "add": [ + { + "metric": "cost_usd", + "n_pairs": 10, + "mean_control": 0.01805025, + "mean_arm": 0.019896370000000003, + "mean_diff": 0.0018461200000000032, + "diff_ci95": { + "lo": 0.0005421500000000051, + "hi": 0.002934329999999999 + }, + "ratio": 1.1022766997687015, + "ratio_ci95": { + "lo": 1.0292981983526395, + "hi": 1.1661606807721805 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 10, + "mean_control": 5212.049999999999, + "mean_arm": 5932.369999999999, + "mean_diff": 720.3199999999997, + "diff_ci95": { + "lo": 286.89000000000016, + "hi": 1076.6200000000013 + }, + "ratio": 1.1382028184687407, + "ratio_ci95": { + "lo": 1.053977459097052, + "hi": 1.2100616556439752 + } + }, + { + "metric": "tool_calls", + "n_pairs": 10, + "mean_control": 2.2, + "mean_arm": 1.8, + "mean_diff": -0.40000000000000013, + "diff_ci95": { + "lo": -0.9, + "hi": 0 + }, + "ratio": 0.8181818181818181, + "ratio_ci95": { + "lo": 0.64, + "hi": 1 + } + }, + { + "metric": "files_read", + "n_pairs": 10, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "impact": [ + { + "metric": "cost_usd", + "n_pairs": 10, + "mean_control": 0.01666486, + "mean_arm": 0.018853339999999996, + "mean_diff": 0.002188479999999996, + "diff_ci95": { + "lo": 0.0014819799999999995, + "hi": 0.0030834799999999996 + }, + "ratio": 1.131323035417039, + "ratio_ci95": { + "lo": 1.0875887661218784, + "hi": 1.188831983537313 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 10, + "mean_control": 4295.46, + "mean_arm": 5115.740000000001, + "mean_diff": 820.2800000000007, + "diff_ci95": { + "lo": 651.7800000000003, + "hi": 1092.5400000000009 + }, + "ratio": 1.1909644135901627, + "ratio_ci95": { + "lo": 1.150744811194738, + "hi": 1.2580500329718127 + } + }, + { + "metric": "tool_calls", + "n_pairs": 10, + "mean_control": 2, + "mean_arm": 1.8, + "mean_diff": -0.19999999999999996, + "diff_ci95": { + "lo": -0.5, + "hi": 0 + }, + "ratio": 0.9, + "ratio_ci95": { + "lo": 0.75, + "hi": 1 + } + }, + { + "metric": "files_read", + "n_pairs": 10, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "locate": [ + { + "metric": "cost_usd", + "n_pairs": 15, + "mean_control": 0.02259353333333333, + "mean_arm": 0.022471146666666667, + "mean_diff": -0.00012238666666666148, + "diff_ci95": { + "lo": -0.0021256066666666645, + "hi": 0.0015869733333333303 + }, + "ratio": 0.994583110801616, + "ratio_ci95": { + "lo": 0.911764916648017, + "hi": 1.0745615487447417 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 15, + "mean_control": 6291.466666666665, + "mean_arm": 6516.213333333333, + "mean_diff": 224.7466666666678, + "diff_ci95": { + "lo": -336.5200000000012, + "hi": 709.7133333333321 + }, + "ratio": 1.0357224600517105, + "ratio_ci95": { + "lo": 0.9497362708660407, + "hi": 1.1184595295661632 + } + }, + { + "metric": "tool_calls", + "n_pairs": 15, + "mean_control": 2.933333333333333, + "mean_arm": 2.6, + "mean_diff": -0.33333333333333304, + "diff_ci95": { + "lo": -0.7333333333333333, + "hi": 0.06666666666666667 + }, + "ratio": 0.8863636363636365, + "ratio_ci95": { + "lo": 0.7608695652173914, + "hi": 1.024390243902439 + } + }, + { + "metric": "files_read", + "n_pairs": 15, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "trace": [ + { + "metric": "cost_usd", + "n_pairs": 15, + "mean_control": 0.015372139999999998, + "mean_arm": 0.017120513333333334, + "mean_diff": 0.0017483733333333359, + "diff_ci95": { + "lo": 0.0005557333333333377, + "hi": 0.002740520000000002 + }, + "ratio": 1.1137364955909415, + "ratio_ci95": { + "lo": 1.033296463961139, + "hi": 1.1905312472692524 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 15, + "mean_control": 4304.139999999999, + "mean_arm": 5081.913333333332, + "mean_diff": 777.7733333333326, + "diff_ci95": { + "lo": 472.86000000000155, + "hi": 1049.3200000000006 + }, + "ratio": 1.1807035396927918, + "ratio_ci95": { + "lo": 1.1022469028454185, + "hi": 1.2558566533003443 + } + }, + { + "metric": "tool_calls", + "n_pairs": 15, + "mean_control": 1.6, + "mean_arm": 1.5333333333333334, + "mean_diff": -0.06666666666666665, + "diff_ci95": { + "lo": -0.4, + "hi": 0.26666666666666666 + }, + "ratio": 0.9583333333333334, + "ratio_ci95": { + "lo": 0.78125, + "hi": 1.1904761904761905 + } + }, + { + "metric": "files_read", + "n_pairs": 15, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ] + } + }, + { + "name": "theauth", + "arm": "stacklit-quiet", + "base": "control", + "n_pairs": 50, + "comparisons": [ + { + "metric": "cost_usd", + "n_pairs": 50, + "mean_control": 0.016739042000000003, + "mean_arm": 0.018990694, + "mean_diff": 0.0022516519999999964, + "diff_ci95": { + "lo": 0.0016020399999999958, + "hi": 0.002903401999999997 + }, + "ratio": 1.1345149859830685, + "ratio_ci95": { + "lo": 1.0940756859030467, + "hi": 1.1761309769453 + } + }, + { + "metric": "total_tokens", + "n_pairs": 50, + "mean_control": 14862.82, + "mean_arm": 19961.24, + "mean_diff": 5098.420000000002, + "diff_ci95": { + "lo": 4031.9, + "hi": 6234.62 + }, + "ratio": 1.3430318068845617, + "ratio_ci95": { + "lo": 1.2693031565143758, + "hi": 1.421933143360037 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 50, + "mean_control": 4638.802, + "mean_arm": 5534.474, + "mean_diff": 895.6720000000005, + "diff_ci95": { + "lo": 706.6780000000034, + "hi": 1094.9260000000004 + }, + "ratio": 1.1930826105533283, + "ratio_ci95": { + "lo": 1.1509931623764478, + "hi": 1.238549241317564 + } + }, + { + "metric": "turns", + "n_pairs": 50, + "mean_control": 2.94, + "mean_arm": 2.86, + "mean_diff": -0.08000000000000007, + "diff_ci95": { + "lo": -0.26, + "hi": 0.1 + }, + "ratio": 0.9727891156462585, + "ratio_ci95": { + "lo": 0.9121621621621622, + "hi": 1.0347222222222223 + } + }, + { + "metric": "tool_calls", + "n_pairs": 50, + "mean_control": 1.94, + "mean_arm": 1.86, + "mean_diff": -0.07999999999999985, + "diff_ci95": { + "lo": -0.26, + "hi": 0.1 + }, + "ratio": 0.9587628865979382, + "ratio_ci95": { + "lo": 0.8703703703703703, + "hi": 1.053763440860215 + } + }, + { + "metric": "read_calls", + "n_pairs": 50, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "grep_glob_calls", + "n_pairs": 50, + "mean_control": 1.94, + "mean_arm": 1.72, + "mean_diff": -0.21999999999999997, + "diff_ci95": { + "lo": -0.42, + "hi": -0.02 + }, + "ratio": 0.8865979381443299, + "ratio_ci95": { + "lo": 0.7909090909090909, + "hi": 0.9891304347826086 + } + }, + { + "metric": "files_read", + "n_pairs": 50, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "read_kb", + "n_pairs": 50, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "seconds", + "n_pairs": 50, + "mean_control": 5.6839197641800006, + "mean_arm": 13.605257256620002, + "mean_diff": 7.921337492440001, + "diff_ci95": { + "lo": 7.390989020820001, + "hi": 8.489940769120002 + }, + "ratio": 2.393639921231855, + "ratio_ci95": { + "lo": 2.2520397785827426, + "hi": 2.540698378242507 + } + }, + { + "metric": "correct", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "recall", + "n_pairs": 35, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "correctness": { + "control": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + } + }, + "correctness_audited": { + "control": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + } + }, + "stacklit_used_in_treatment": { + "k": 0, + "n": 50, + "rate": 0, + "wilson95": { + "lo": 0, + "hi": 0.07135003417431873 + } + }, + "hook_fired_in_treatment": { + "k": 50, + "n": 50, + "rate": 1, + "wilson95": { + "lo": 0.9286499658256813, + "hi": 1 + } + }, + "by_kind": { + "add": [ + { + "metric": "cost_usd", + "n_pairs": 10, + "mean_control": 0.01962268, + "mean_arm": 0.021608850000000006, + "mean_diff": 0.001986170000000006, + "diff_ci95": { + "lo": 0.0004344500000000029, + "hi": 0.0033263399999999997 + }, + "ratio": 1.1012180803029967, + "ratio_ci95": { + "lo": 1.0223505964617685, + "hi": 1.1697124458291999 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 10, + "mean_control": 5525.08, + "mean_arm": 6416.950000000002, + "mean_diff": 891.8700000000017, + "diff_ci95": { + "lo": 445.9800000000003, + "hi": 1286.8500000000022 + }, + "ratio": 1.1614220970556086, + "ratio_ci95": { + "lo": 1.0814873124652715, + "hi": 1.2298204369933834 + } + }, + { + "metric": "tool_calls", + "n_pairs": 10, + "mean_control": 3.1, + "mean_arm": 3.1, + "mean_diff": 0, + "diff_ci95": { + "lo": -0.6, + "hi": 0.5 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 0.8235294117647058, + "hi": 1.1785714285714286 + } + }, + { + "metric": "files_read", + "n_pairs": 10, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "impact": [ + { + "metric": "cost_usd", + "n_pairs": 10, + "mean_control": 0.01571698, + "mean_arm": 0.01886232, + "mean_diff": 0.00314534, + "diff_ci95": { + "lo": 0.0025197000000000023, + "hi": 0.004015390000000002 + }, + "ratio": 1.200123687884059, + "ratio_ci95": { + "lo": 1.1608796623362612, + "hi": 1.254133074771508 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 10, + "mean_control": 4218.4800000000005, + "mean_arm": 5261.92, + "mean_diff": 1043.4399999999996, + "diff_ci95": { + "lo": 824.519999999999, + "hi": 1359.009999999999 + }, + "ratio": 1.2473497563103297, + "ratio_ci95": { + "lo": 1.195100045851394, + "hi": 1.322704245451302 + } + }, + { + "metric": "tool_calls", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1.2, + "mean_diff": 0.19999999999999996, + "diff_ci95": { + "lo": 0, + "hi": 0.5 + }, + "ratio": 1.2, + "ratio_ci95": { + "lo": 1, + "hi": 1.5 + } + }, + { + "metric": "files_read", + "n_pairs": 10, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 10, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "locate": [ + { + "metric": "cost_usd", + "n_pairs": 15, + "mean_control": 0.015509326666666667, + "mean_arm": 0.018502933333333336, + "mean_diff": 0.002993606666666669, + "diff_ci95": { + "lo": 0.0018448399999999995, + "hi": 0.004460746666666667 + }, + "ratio": 1.1930197700394474, + "ratio_ci95": { + "lo": 1.1186585446375814, + "hi": 1.2863890167929812 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 15, + "mean_control": 4417.126666666667, + "mean_arm": 5463.799999999999, + "mean_diff": 1046.6733333333323, + "diff_ci95": { + "lo": 664.5333333333314, + "hi": 1510.7933333333349 + }, + "ratio": 1.2369579621141342, + "ratio_ci95": { + "lo": 1.1491375616102202, + "hi": 1.3421661516613064 + } + }, + { + "metric": "tool_calls", + "n_pairs": 15, + "mean_control": 1.6, + "mean_arm": 1.6666666666666667, + "mean_diff": 0.06666666666666665, + "diff_ci95": { + "lo": -0.2, + "hi": 0.3333333333333333 + }, + "ratio": 1.0416666666666667, + "ratio_ci95": { + "lo": 0.8695652173913043, + "hi": 1.2380952380952381 + } + }, + { + "metric": "files_read", + "n_pairs": 15, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "trace": [ + { + "metric": "cost_usd", + "n_pairs": 15, + "mean_control": 0.016727706666666665, + "mean_arm": 0.0178186, + "mean_diff": 0.0010908933333333357, + "diff_ci95": { + "lo": 0.00012209999999999906, + "hi": 0.002025206666666668 + }, + "ratio": 1.065214757472234, + "ratio_ci95": { + "lo": 1.006868270523399, + "hi": 1.1294917791994759 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 15, + "mean_control": 4549.84, + "mean_arm": 5198.533333333332, + "mean_diff": 648.6933333333318, + "diff_ci95": { + "lo": 344.80666666666804, + "hi": 960.9333333333324 + }, + "ratio": 1.1425749769955278, + "ratio_ci95": { + "lo": 1.0719707599630548, + "hi": 1.2219587467967596 + } + }, + { + "metric": "tool_calls", + "n_pairs": 15, + "mean_control": 2.1333333333333333, + "mean_arm": 1.6666666666666667, + "mean_diff": -0.46666666666666656, + "diff_ci95": { + "lo": -0.7333333333333333, + "hi": -0.2 + }, + "ratio": 0.78125, + "ratio_ci95": { + "lo": 0.6666666666666666, + "hi": 0.9 + } + }, + { + "metric": "files_read", + "n_pairs": 15, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 15, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ] + } + }, + { + "name": "pooled", + "arm": "stacklit-quiet", + "base": "control", + "n_pairs": 150, + "comparisons": [ + { + "metric": "cost_usd", + "n_pairs": 150, + "mean_control": 0.019081443333333333, + "mean_arm": 0.020580145333333327, + "mean_diff": 0.0014987019999999941, + "diff_ci95": { + "lo": 0.0010254539999999975, + "hi": 0.0019740013333333344 + }, + "ratio": 1.0785423814026645, + "ratio_ci95": { + "lo": 1.052243054040162, + "hi": 1.1067047145935445 + } + }, + { + "metric": "total_tokens", + "n_pairs": 150, + "mean_control": 16924.673333333332, + "mean_arm": 21709.673333333332, + "mean_diff": 4785, + "diff_ci95": { + "lo": 3928.74, + "hi": 5662.966666666666 + }, + "ratio": 1.2827233297658922, + "ratio_ci95": { + "lo": 1.2277852760300136, + "hi": 1.3413003030014201 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 150, + "mean_control": 5251.763333333337, + "mean_arm": 5974.205333333334, + "mean_diff": 722.4419999999973, + "diff_ci95": { + "lo": 582.3833333333341, + "hi": 864.8260000000009 + }, + "ratio": 1.1375617967044713, + "ratio_ci95": { + "lo": 1.107972558962997, + "hi": 1.1688050729528359 + } + }, + { + "metric": "turns", + "n_pairs": 150, + "mean_control": 3.1933333333333334, + "mean_arm": 3.0866666666666664, + "mean_diff": -0.10666666666666691, + "diff_ci95": { + "lo": -0.25333333333333335, + "hi": 0.04 + }, + "ratio": 0.9665970772442588, + "ratio_ci95": { + "lo": 0.9235412474849095, + "hi": 1.0125 + } + }, + { + "metric": "tool_calls", + "n_pairs": 150, + "mean_control": 2.1933333333333334, + "mean_arm": 2.0866666666666664, + "mean_diff": -0.10666666666666691, + "diff_ci95": { + "lo": -0.25333333333333335, + "hi": 0.03333333333333333 + }, + "ratio": 0.9513677811550151, + "ratio_ci95": { + "lo": 0.8901408450704226, + "hi": 1.0163398692810457 + } + }, + { + "metric": "read_calls", + "n_pairs": 150, + "mean_control": 0.02, + "mean_arm": 0.006666666666666667, + "mean_diff": -0.013333333333333332, + "diff_ci95": { + "lo": -0.04, + "hi": 0.013333333333333334 + }, + "ratio": 0.33333333333333337, + "ratio_ci95": { + "lo": 0, + "hi": 2 + } + }, + { + "metric": "grep_glob_calls", + "n_pairs": 150, + "mean_control": 2.1733333333333333, + "mean_arm": 1.9933333333333334, + "mean_diff": -0.17999999999999994, + "diff_ci95": { + "lo": -0.3333333333333333, + "hi": -0.02666666666666667 + }, + "ratio": 0.9171779141104295, + "ratio_ci95": { + "lo": 0.8540372670807453, + "hi": 0.9862068965517241 + } + }, + { + "metric": "files_read", + "n_pairs": 150, + "mean_control": 0.02, + "mean_arm": 0.006666666666666667, + "mean_diff": -0.013333333333333332, + "diff_ci95": { + "lo": -0.04, + "hi": 0.013333333333333334 + }, + "ratio": 0.33333333333333337, + "ratio_ci95": { + "lo": 0, + "hi": 2 + } + }, + { + "metric": "read_kb", + "n_pairs": 150, + "mean_control": 0.11759114583333333, + "mean_arm": 0.0045377604166666665, + "mean_diff": -0.11305338541666667, + "diff_ci95": { + "lo": -0.28009765625, + "hi": 0.0045377604166666665 + }, + "ratio": 0.03858930351013177, + "ratio_ci95": { + "lo": 0, + "hi": 0.27527646129541866 + } + }, + { + "metric": "seconds", + "n_pairs": 150, + "mean_control": 6.0273182922999995, + "mean_arm": 9.265069095520005, + "mean_diff": 3.2377508032200053, + "diff_ci95": { + "lo": 2.633059788213331, + "hi": 3.853211420393337 + }, + "ratio": 1.5371793302099686, + "ratio_ci95": { + "lo": 1.4249145516502983, + "hi": 1.653671318817187 + } + }, + { + "metric": "correct", + "n_pairs": 150, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 150, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "recall", + "n_pairs": 100, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "correctness": { + "control": { + "k": 150, + "n": 150, + "rate": 1, + "wilson95": { + "lo": 0.9750288608542812, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 150, + "n": 150, + "rate": 1, + "wilson95": { + "lo": 0.9750288608542812, + "hi": 1 + } + } + }, + "correctness_audited": { + "control": { + "k": 150, + "n": 150, + "rate": 1, + "wilson95": { + "lo": 0.9750288608542812, + "hi": 1 + } + }, + "stacklit-quiet": { + "k": 150, + "n": 150, + "rate": 1, + "wilson95": { + "lo": 0.9750288608542812, + "hi": 1 + } + } + }, + "stacklit_used_in_treatment": { + "k": 0, + "n": 150, + "rate": 0, + "wilson95": { + "lo": 1.734723475976807e-18, + "hi": 0.02497113914571871 + } + }, + "hook_fired_in_treatment": { + "k": 150, + "n": 150, + "rate": 1, + "wilson95": { + "lo": 0.9750288608542812, + "hi": 1 + } + }, + "by_kind": { + "add": [ + { + "metric": "cost_usd", + "n_pairs": 30, + "mean_control": 0.019664516666666663, + "mean_arm": 0.021815503333333333, + "mean_diff": 0.0021509866666666703, + "diff_ci95": { + "lo": 0.0010923133333333426, + "hi": 0.003221840000000002 + }, + "ratio": 1.109384161488841, + "ratio_ci95": { + "lo": 1.0545739648002532, + "hi": 1.1665672008916002 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 30, + "mean_control": 5539.116666666667, + "mean_arm": 6423.3033333333315, + "mean_diff": 884.1866666666647, + "diff_ci95": { + "lo": 558.2166666666637, + "hi": 1217.949999999998 + }, + "ratio": 1.1596259331361496, + "ratio_ci95": { + "lo": 1.0987329530012493, + "hi": 1.2229620935505203 + } + }, + { + "metric": "tool_calls", + "n_pairs": 30, + "mean_control": 2.6666666666666665, + "mean_arm": 2.566666666666667, + "mean_diff": -0.09999999999999964, + "diff_ci95": { + "lo": -0.4666666666666667, + "hi": 0.26666666666666666 + }, + "ratio": 0.9625000000000001, + "ratio_ci95": { + "lo": 0.8352941176470589, + "hi": 1.1111111111111112 + } + }, + { + "metric": "files_read", + "n_pairs": 30, + "mean_control": 0, + "mean_arm": 0.03333333333333333, + "mean_diff": 0.03333333333333333, + "diff_ci95": { + "lo": 0, + "hi": 0.1 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 30, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 30, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "impact": [ + { + "metric": "cost_usd", + "n_pairs": 30, + "mean_control": 0.015256010000000007, + "mean_arm": 0.01783121, + "mean_diff": 0.0025751999999999928, + "diff_ci95": { + "lo": 0.0022046633333333344, + "hi": 0.0030104766666666652 + }, + "ratio": 1.1687990503414714, + "ratio_ci95": { + "lo": 1.1430786431409983, + "hi": 1.198314028889585 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 30, + "mean_control": 4070.409999999999, + "mean_arm": 4954.71, + "mean_diff": 884.3000000000011, + "diff_ci95": { + "lo": 775.2966666666674, + "hi": 1021.6900000000013 + }, + "ratio": 1.2172508420527666, + "ratio_ci95": { + "lo": 1.1902831269078566, + "hi": 1.2511524442094342 + } + }, + { + "metric": "tool_calls", + "n_pairs": 30, + "mean_control": 1.3333333333333333, + "mean_arm": 1.3333333333333333, + "mean_diff": 0, + "diff_ci95": { + "lo": -0.13333333333333333, + "hi": 0.13333333333333333 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 0.9024390243902439, + "hi": 1.105263157894737 + } + }, + { + "metric": "files_read", + "n_pairs": 30, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 30, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 30, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "locate": [ + { + "metric": "cost_usd", + "n_pairs": 50, + "mean_control": 0.023130792, + "mean_arm": 0.02348841, + "mean_diff": 0.0003576180000000005, + "diff_ci95": { + "lo": -0.0007619519999999902, + "hi": 0.0014234999999999999 + }, + "ratio": 1.01546068980258, + "ratio_ci95": { + "lo": 0.968782958873217, + "hi": 1.0643542746975319 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 50, + "mean_control": 6371.272000000001, + "mean_arm": 6804.110000000002, + "mean_diff": 432.83800000000156, + "diff_ci95": { + "lo": 117.3340000000014, + "hi": 735.6960000000032 + }, + "ratio": 1.0679358847024585, + "ratio_ci95": { + "lo": 1.0177045652987398, + "hi": 1.1205415823013891 + } + }, + { + "metric": "tool_calls", + "n_pairs": 50, + "mean_control": 2.78, + "mean_arm": 2.56, + "mean_diff": -0.21999999999999975, + "diff_ci95": { + "lo": -0.54, + "hi": 0.08 + }, + "ratio": 0.920863309352518, + "ratio_ci95": { + "lo": 0.821917808219178, + "hi": 1.0344827586206897 + } + }, + { + "metric": "files_read", + "n_pairs": 50, + "mean_control": 0.06, + "mean_arm": 0, + "mean_diff": -0.06, + "diff_ci95": { + "lo": -0.14, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 50, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ], + "trace": [ + { + "metric": "cost_usd", + "n_pairs": 40, + "mean_control": 0.0164515275, + "mean_arm": 0.018079997500000007, + "mean_diff": 0.0016284700000000069, + "diff_ci95": { + "lo": 0.0009496374999999974, + "hi": 0.0022510050000000116 + }, + "ratio": 1.0989859452260593, + "ratio_ci95": { + "lo": 1.055257994394205, + "hi": 1.1426054181856906 + } + }, + { + "metric": "uncached_eq_tokens", + "n_pairs": 40, + "mean_control": 4522.877499999998, + "mean_arm": 5364.6224999999995, + "mean_diff": 841.7450000000017, + "diff_ci95": { + "lo": 615.9150000000024, + "hi": 1078.1500000000015 + }, + "ratio": 1.1861082905738662, + "ratio_ci95": { + "lo": 1.131338236160447, + "hi": 1.2452463466304444 + } + }, + { + "metric": "tool_calls", + "n_pairs": 40, + "mean_control": 1.75, + "mean_arm": 1.7, + "mean_diff": -0.050000000000000044, + "diff_ci95": { + "lo": -0.275, + "hi": 0.175 + }, + "ratio": 0.9714285714285714, + "ratio_ci95": { + "lo": 0.8529411764705882, + "hi": 1.109375 + } + }, + { + "metric": "files_read", + "n_pairs": 40, + "mean_control": 0, + "mean_arm": 0, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 0, + "ratio_ci95": { + "lo": 0, + "hi": 0 + } + }, + { + "metric": "correct", + "n_pairs": 40, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + }, + { + "metric": "correct_audited", + "n_pairs": 40, + "mean_control": 1, + "mean_arm": 1, + "mean_diff": 0, + "diff_ci95": { + "lo": 0, + "hi": 0 + }, + "ratio": 1, + "ratio_ci95": { + "lo": 1, + "hi": 1 + } + } + ] + } + } + ], + "metric_definitions": { + "correct": "answer names the expected paths (0 or 1)", + "correct_audited": "correct after the three documented grading fixes (see RESULTS.md)", + "cost_usd": "notional cost, USD", + "files_read": "distinct files opened with Read", + "grep_glob_calls": "Grep + Glob calls", + "read_calls": "Read calls", + "read_kb": "KB returned by Read", + "recall": "fraction of all_of paths named (tasks with all_of only)", + "seconds": "wall seconds (includes clone and stacklit setup)", + "tool_calls": "all tool calls", + "total_tokens": "input + output + cache creation + cache read", + "turns": "model turns", + "uncached_eq_tokens": "input + output + cache creation + 0.1 x cache read" + } +} \ No newline at end of file diff --git a/bench/agent-ab/round3/tables.md b/bench/agent-ab/round3/tables.md new file mode 100644 index 0000000..99b9c1c --- /dev/null +++ b/bench/agent-ab/round3/tables.md @@ -0,0 +1,206 @@ +Model: claude-sonnet-5-5. Bootstrap: 10000 resamples, seed 20260508, paired by task and trial. Ledger total: 5.95 USD notional. + +### theauth-go: control vs stacklit-quiet (50 pairs) + +| metric | n | control | stacklit-quiet | diff (arm minus control) [95% CI] | ratio (arm / control) [95% CI] | +|---|---|---|---|---|---| +| cost_usd | 50 | 0.0222 | 0.0231 | 0.0009 [-0.0001, 0.0019] | 1.04 [1.00, 1.09] | +| total_tokens | 50 | 19116 | 24673 | 5557 [3537, 7652] | 1.29 [1.18, 1.42] | +| uncached_eq_tokens | 50 | 6036 | 6699 | 663 [343, 979] | 1.11 [1.05, 1.17] | +| turns | 50 | 3.44 | 3.44 | 0.00 [-0.34, 0.34] | 1.00 [0.91, 1.10] | +| tool_calls | 50 | 2.44 | 2.44 | 0.00 [-0.34, 0.34] | 1.00 [0.87, 1.16] | +| read_calls | 50 | 0.06 | 0.02 | -0.04 [-0.12, 0.04] | 0.33 [0.00, 2.00] | +| grep_glob_calls | 50 | 2.38 | 2.36 | -0.02 [-0.38, 0.34] | 0.99 [0.86, 1.15] | +| files_read | 50 | 0.06 | 0.02 | -0.04 [-0.12, 0.04] | 0.33 [0.00, 2.00] | +| read_kb | 50 | 0.35 | 0.01 | -0.34 [-0.80, 0.01] | 0.04 [0.00, 0.27] | +| seconds | 50 | 7.01 | 7.39 | 0.38 [-0.21, 0.98] | 1.05 [0.97, 1.15] | +| correct | 50 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| correct_audited | 50 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| recall | 30 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | + +Correctness after audit fixes: control 50/50 = 100.0% [92.9, 100.0]; stacklit-quiet 50/50 = 100.0% [92.9, 100.0]. + +Correctness (pre-registered grader): control 50/50 = 100.0% [92.9, 100.0]; stacklit-quiet 50/50 = 100.0% [92.9, 100.0]. Stacklit commands used in 0/50 = 0.0% [0.0, 7.1] of treatment sessions; session hook fired in 50/50 = 100.0% [92.9, 100.0]. + +By task kind (theauth-go): + +| kind | metric | n | control | stacklit-quiet | diff [95% CI] | ratio [95% CI] | +|---|---|---|---|---|---|---| +| add | cost_usd | 10 | 0.0213 | 0.0239 | 0.0026 [0.0002, 0.0051] | 1.12 [1.01, 1.25] | +| add | uncached_eq_tokens | 10 | 5880 | 6921 | 1040 [278, 1822] | 1.18 [1.04, 1.33] | +| add | tool_calls | 10 | 2.70 | 2.80 | 0.10 [-0.70, 1.00] | 1.04 [0.77, 1.45] | +| add | files_read | 10 | 0.00 | 0.10 | 0.10 [0.00, 0.30] | 0.00 [0.00, 0.00] | +| add | correct | 10 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| add | correct_audited | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| impact | cost_usd | 10 | 0.0134 | 0.0158 | 0.0024 [0.0023, 0.0025] | 1.18 [1.17, 1.19] | +| impact | uncached_eq_tokens | 10 | 3697 | 4486 | 789 [772, 805] | 1.21 [1.21, 1.22] | +| impact | tool_calls | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| impact | files_read | 10 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| impact | correct | 10 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| impact | correct_audited | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| locate | cost_usd | 20 | 0.0292 | 0.0280 | -0.0013 [-0.0030, 0.0005] | 0.96 [0.90, 1.02] | +| locate | uncached_eq_tokens | 20 | 7897 | 8025 | 129 [-389, 635] | 1.02 [0.95, 1.08] | +| locate | tool_calls | 20 | 3.55 | 3.20 | -0.35 [-1.00, 0.30] | 0.90 [0.74, 1.10] | +| locate | files_read | 20 | 0.15 | 0.00 | -0.15 [-0.30, 0.00] | 0.00 [0.00, 0.00] | +| locate | correct | 20 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| locate | correct_audited | 20 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| trace | cost_usd | 10 | 0.0177 | 0.0199 | 0.0023 [0.0010, 0.0036] | 1.13 [1.05, 1.22] | +| trace | uncached_eq_tokens | 10 | 4811 | 6038 | 1227 [628, 1859] | 1.26 [1.12, 1.42] | +| trace | tool_calls | 10 | 1.40 | 2.00 | 0.60 [0.20, 1.00] | 1.43 [1.17, 1.71] | +| trace | files_read | 10 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| trace | correct | 10 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| trace | correct_audited | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | + +### featuredrop: control vs stacklit-quiet (50 pairs) + +| metric | n | control | stacklit-quiet | diff (arm minus control) [95% CI] | ratio (arm / control) [95% CI] | +|---|---|---|---|---|---| +| cost_usd | 50 | 0.0183 | 0.0196 | 0.0013 [0.0005, 0.0020] | 1.07 [1.02, 1.11] | +| total_tokens | 50 | 16795 | 20495 | 3700 [2617, 4739] | 1.22 [1.15, 1.29] | +| uncached_eq_tokens | 50 | 5080 | 5689 | 609 [379, 805] | 1.12 [1.07, 1.16] | +| turns | 50 | 3.20 | 2.96 | -0.24 [-0.42, -0.06] | 0.92 [0.87, 0.98] | +| tool_calls | 50 | 2.20 | 1.96 | -0.24 [-0.44, -0.06] | 0.89 [0.81, 0.97] | +| read_calls | 50 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| grep_glob_calls | 50 | 2.20 | 1.90 | -0.30 [-0.50, -0.12] | 0.86 [0.79, 0.94] | +| files_read | 50 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| read_kb | 50 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| seconds | 50 | 5.39 | 6.80 | 1.41 [1.02, 1.76] | 1.26 [1.18, 1.34] | +| correct | 50 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| correct_audited | 50 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| recall | 35 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | + +Correctness after audit fixes: control 50/50 = 100.0% [92.9, 100.0]; stacklit-quiet 50/50 = 100.0% [92.9, 100.0]. + +Correctness (pre-registered grader): control 50/50 = 100.0% [92.9, 100.0]; stacklit-quiet 50/50 = 100.0% [92.9, 100.0]. Stacklit commands used in 0/50 = 0.0% [0.0, 7.1] of treatment sessions; session hook fired in 50/50 = 100.0% [92.9, 100.0]. + +By task kind (featuredrop): + +| kind | metric | n | control | stacklit-quiet | diff [95% CI] | ratio [95% CI] | +|---|---|---|---|---|---|---| +| add | cost_usd | 10 | 0.0181 | 0.0199 | 0.0018 [0.0005, 0.0029] | 1.10 [1.03, 1.17] | +| add | uncached_eq_tokens | 10 | 5212 | 5932 | 720 [287, 1077] | 1.14 [1.05, 1.21] | +| add | tool_calls | 10 | 2.20 | 1.80 | -0.40 [-0.90, 0.00] | 0.82 [0.64, 1.00] | +| add | files_read | 10 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| add | correct | 10 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| add | correct_audited | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| impact | cost_usd | 10 | 0.0167 | 0.0189 | 0.0022 [0.0015, 0.0031] | 1.13 [1.09, 1.19] | +| impact | uncached_eq_tokens | 10 | 4295 | 5116 | 820 [652, 1093] | 1.19 [1.15, 1.26] | +| impact | tool_calls | 10 | 2.00 | 1.80 | -0.20 [-0.50, 0.00] | 0.90 [0.75, 1.00] | +| impact | files_read | 10 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| impact | correct | 10 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| impact | correct_audited | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| locate | cost_usd | 15 | 0.0226 | 0.0225 | -0.0001 [-0.0021, 0.0016] | 0.99 [0.91, 1.07] | +| locate | uncached_eq_tokens | 15 | 6291 | 6516 | 225 [-337, 710] | 1.04 [0.95, 1.12] | +| locate | tool_calls | 15 | 2.93 | 2.60 | -0.33 [-0.73, 0.07] | 0.89 [0.76, 1.02] | +| locate | files_read | 15 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| locate | correct | 15 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| locate | correct_audited | 15 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| trace | cost_usd | 15 | 0.0154 | 0.0171 | 0.0017 [0.0006, 0.0027] | 1.11 [1.03, 1.19] | +| trace | uncached_eq_tokens | 15 | 4304 | 5082 | 778 [473, 1049] | 1.18 [1.10, 1.26] | +| trace | tool_calls | 15 | 1.60 | 1.53 | -0.07 [-0.40, 0.27] | 0.96 [0.78, 1.19] | +| trace | files_read | 15 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| trace | correct | 15 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| trace | correct_audited | 15 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | + +### theauth: control vs stacklit-quiet (50 pairs) + +| metric | n | control | stacklit-quiet | diff (arm minus control) [95% CI] | ratio (arm / control) [95% CI] | +|---|---|---|---|---|---| +| cost_usd | 50 | 0.0167 | 0.0190 | 0.0023 [0.0016, 0.0029] | 1.13 [1.09, 1.18] | +| total_tokens | 50 | 14863 | 19961 | 5098 [4032, 6235] | 1.34 [1.27, 1.42] | +| uncached_eq_tokens | 50 | 4639 | 5534 | 896 [707, 1095] | 1.19 [1.15, 1.24] | +| turns | 50 | 2.94 | 2.86 | -0.08 [-0.26, 0.10] | 0.97 [0.91, 1.03] | +| tool_calls | 50 | 1.94 | 1.86 | -0.08 [-0.26, 0.10] | 0.96 [0.87, 1.05] | +| read_calls | 50 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| grep_glob_calls | 50 | 1.94 | 1.72 | -0.22 [-0.42, -0.02] | 0.89 [0.79, 0.99] | +| files_read | 50 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| read_kb | 50 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| seconds | 50 | 5.68 | 13.61 | 7.92 [7.39, 8.49] | 2.39 [2.25, 2.54] | +| correct | 50 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| correct_audited | 50 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| recall | 35 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | + +Correctness after audit fixes: control 50/50 = 100.0% [92.9, 100.0]; stacklit-quiet 50/50 = 100.0% [92.9, 100.0]. + +Correctness (pre-registered grader): control 50/50 = 100.0% [92.9, 100.0]; stacklit-quiet 50/50 = 100.0% [92.9, 100.0]. Stacklit commands used in 0/50 = 0.0% [0.0, 7.1] of treatment sessions; session hook fired in 50/50 = 100.0% [92.9, 100.0]. + +By task kind (theauth): + +| kind | metric | n | control | stacklit-quiet | diff [95% CI] | ratio [95% CI] | +|---|---|---|---|---|---|---| +| add | cost_usd | 10 | 0.0196 | 0.0216 | 0.0020 [0.0004, 0.0033] | 1.10 [1.02, 1.17] | +| add | uncached_eq_tokens | 10 | 5525 | 6417 | 892 [446, 1287] | 1.16 [1.08, 1.23] | +| add | tool_calls | 10 | 3.10 | 3.10 | 0.00 [-0.60, 0.50] | 1.00 [0.82, 1.18] | +| add | files_read | 10 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| add | correct | 10 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| add | correct_audited | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| impact | cost_usd | 10 | 0.0157 | 0.0189 | 0.0031 [0.0025, 0.0040] | 1.20 [1.16, 1.25] | +| impact | uncached_eq_tokens | 10 | 4218 | 5262 | 1043 [825, 1359] | 1.25 [1.20, 1.32] | +| impact | tool_calls | 10 | 1.00 | 1.20 | 0.20 [0.00, 0.50] | 1.20 [1.00, 1.50] | +| impact | files_read | 10 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| impact | correct | 10 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| impact | correct_audited | 10 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| locate | cost_usd | 15 | 0.0155 | 0.0185 | 0.0030 [0.0018, 0.0045] | 1.19 [1.12, 1.29] | +| locate | uncached_eq_tokens | 15 | 4417 | 5464 | 1047 [665, 1511] | 1.24 [1.15, 1.34] | +| locate | tool_calls | 15 | 1.60 | 1.67 | 0.07 [-0.20, 0.33] | 1.04 [0.87, 1.24] | +| locate | files_read | 15 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| locate | correct | 15 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| locate | correct_audited | 15 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| trace | cost_usd | 15 | 0.0167 | 0.0178 | 0.0011 [0.0001, 0.0020] | 1.07 [1.01, 1.13] | +| trace | uncached_eq_tokens | 15 | 4550 | 5199 | 649 [345, 961] | 1.14 [1.07, 1.22] | +| trace | tool_calls | 15 | 2.13 | 1.67 | -0.47 [-0.73, -0.20] | 0.78 [0.67, 0.90] | +| trace | files_read | 15 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| trace | correct | 15 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| trace | correct_audited | 15 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | + +### pooled: control vs stacklit-quiet (150 pairs) + +| metric | n | control | stacklit-quiet | diff (arm minus control) [95% CI] | ratio (arm / control) [95% CI] | +|---|---|---|---|---|---| +| cost_usd | 150 | 0.0191 | 0.0206 | 0.0015 [0.0010, 0.0020] | 1.08 [1.05, 1.11] | +| total_tokens | 150 | 16925 | 21710 | 4785 [3929, 5663] | 1.28 [1.23, 1.34] | +| uncached_eq_tokens | 150 | 5252 | 5974 | 722 [582, 865] | 1.14 [1.11, 1.17] | +| turns | 150 | 3.19 | 3.09 | -0.11 [-0.25, 0.04] | 0.97 [0.92, 1.01] | +| tool_calls | 150 | 2.19 | 2.09 | -0.11 [-0.25, 0.03] | 0.95 [0.89, 1.02] | +| read_calls | 150 | 0.02 | 0.01 | -0.01 [-0.04, 0.01] | 0.33 [0.00, 2.00] | +| grep_glob_calls | 150 | 2.17 | 1.99 | -0.18 [-0.33, -0.03] | 0.92 [0.85, 0.99] | +| files_read | 150 | 0.02 | 0.01 | -0.01 [-0.04, 0.01] | 0.33 [0.00, 2.00] | +| read_kb | 150 | 0.12 | 0.00 | -0.11 [-0.28, 0.00] | 0.04 [0.00, 0.28] | +| seconds | 150 | 6.03 | 9.27 | 3.24 [2.63, 3.85] | 1.54 [1.42, 1.65] | +| correct | 150 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| correct_audited | 150 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| recall | 100 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | + +Correctness after audit fixes: control 150/150 = 100.0% [97.5, 100.0]; stacklit-quiet 150/150 = 100.0% [97.5, 100.0]. + +Correctness (pre-registered grader): control 150/150 = 100.0% [97.5, 100.0]; stacklit-quiet 150/150 = 100.0% [97.5, 100.0]. Stacklit commands used in 0/150 = 0.0% [0.0, 2.5] of treatment sessions; session hook fired in 150/150 = 100.0% [97.5, 100.0]. + +By task kind (pooled): + +| kind | metric | n | control | stacklit-quiet | diff [95% CI] | ratio [95% CI] | +|---|---|---|---|---|---|---| +| add | cost_usd | 30 | 0.0197 | 0.0218 | 0.0022 [0.0011, 0.0032] | 1.11 [1.05, 1.17] | +| add | uncached_eq_tokens | 30 | 5539 | 6423 | 884 [558, 1218] | 1.16 [1.10, 1.22] | +| add | tool_calls | 30 | 2.67 | 2.57 | -0.10 [-0.47, 0.27] | 0.96 [0.84, 1.11] | +| add | files_read | 30 | 0.00 | 0.03 | 0.03 [0.00, 0.10] | 0.00 [0.00, 0.00] | +| add | correct | 30 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| add | correct_audited | 30 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| impact | cost_usd | 30 | 0.0153 | 0.0178 | 0.0026 [0.0022, 0.0030] | 1.17 [1.14, 1.20] | +| impact | uncached_eq_tokens | 30 | 4070 | 4955 | 884 [775, 1022] | 1.22 [1.19, 1.25] | +| impact | tool_calls | 30 | 1.33 | 1.33 | 0.00 [-0.13, 0.13] | 1.00 [0.90, 1.11] | +| impact | files_read | 30 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| impact | correct | 30 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| impact | correct_audited | 30 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| locate | cost_usd | 50 | 0.0231 | 0.0235 | 0.0004 [-0.0008, 0.0014] | 1.02 [0.97, 1.06] | +| locate | uncached_eq_tokens | 50 | 6371 | 6804 | 433 [117, 736] | 1.07 [1.02, 1.12] | +| locate | tool_calls | 50 | 2.78 | 2.56 | -0.22 [-0.54, 0.08] | 0.92 [0.82, 1.03] | +| locate | files_read | 50 | 0.06 | 0.00 | -0.06 [-0.14, 0.00] | 0.00 [0.00, 0.00] | +| locate | correct | 50 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| locate | correct_audited | 50 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +| trace | cost_usd | 40 | 0.0165 | 0.0181 | 0.0016 [0.0009, 0.0023] | 1.10 [1.06, 1.14] | +| trace | uncached_eq_tokens | 40 | 4523 | 5365 | 842 [616, 1078] | 1.19 [1.13, 1.25] | +| trace | tool_calls | 40 | 1.75 | 1.70 | -0.05 [-0.28, 0.17] | 0.97 [0.85, 1.11] | +| trace | files_read | 40 | 0.00 | 0.00 | 0.00 [0.00, 0.00] | 0.00 [0.00, 0.00] | +| trace | correct | 40 | 1.000 | 1.000 | 0.000 [0.000, 0.000] | 1.00 [1.00, 1.00] | +| trace | correct_audited | 40 | 1.00 | 1.00 | 0.00 [0.00, 0.00] | 1.00 [1.00, 1.00] | +