docs: publish round 3 of the agent test, quiet mode does not remove the cost

CHANGELOG.md

@@ -11,7 +11,7 @@ Follows the first real-agent A/B (bench/agent-ab, pull request #77), which found
 - `query` adds one line naming the next ranked files and the `explain` and `path` commands when the question words suggest a flow or chain, or the top files are scattered with flat scores. It stays inside the token budget and leaves single-answer queries unchanged. `STACKLIT_QUERY_NEXT` (0 turns it off), `STACKLIT_FLOW_WORDS`, `STACKLIT_QUERY_SPREAD_DIRS`, `STACKLIT_QUERY_SPREAD_TOP`, `STACKLIT_QUERY_FLATNESS`.
 - `impact` on a file or directory lists its importers as `path:line`, with the sources checked for script languages so the list matches a grep, and `--calls` for the previous call-level view. A Go package directory used to be an error. About 40 to 160 tokens for a package against 110 to 390 for `grep -rn` of the same import. `STACKLIT_IMPACT_SCAN`, `STACKLIT_IMPACT_SCAN_MAX`.
 - Small repo mode: below `small_repo_files` (default 600, `STACKLIT_SMALL_REPO_FILES`, 0 off) source files, the session hook, subagent hook and `setup` carry a one line note that grep is enough instead of the digest (about 60 tokens instead of 490).
-- Quiet repo mode: below `quiet_repo_files` (default 5000, `STACKLIT_QUIET_REPO_FILES`, 0 off) source files, the session and subagent hooks inject nothing at all. Round 2 measured Stacklit as costlier at 412 to 2,919 files and cheaper only at 15,978, so the 5000 default sits between and is not yet measured. The `setup` rules block stays the one line note.
+- Quiet repo mode: below `quiet_repo_files` (default 5000, `STACKLIT_QUIET_REPO_FILES`, 0 off) source files, the session and subagent hooks inject nothing at all. Round 2 measured Stacklit as costlier at 412 to 2,919 files and cheaper only at 15,978, so the 5000 default sits between and is not yet measured. The `setup` rules block stays the one line note. Round 3 (`bench/agent-ab/RESULTS-round3.md`) measured it: below 5,000 files a quiet session still cost 1.08 times a session without Stacklit [1.05, 1.11], against 1.05 to 1.42 before, so it does not remove the cost.
 - `impact` prints a `trust:` line saying how many listed dependents are sure (score 0.9 or more) and how many need a confirming grep.
 - `path` adds advice after the weakest hop score: sure (0.9 or more) needs no confirming grep, otherwise grep the weak hop.
 - A pre-registered second A/B round in `bench/agent-ab/tasks-round2/`, with a `stacklit-e` arm in the harness. It has not been run.

bench/agent-ab/RESULTS-round3.md

@@ -0,0 +1,40 @@
+# Round 3 results: does quiet repo mode remove the cost below 5,000 files?
+
+Pre-registration: `PREREGISTRATION-round3.md`. Raw records: `results-round3/`; ledger, summary and tables: `round3/`. Model claude-sonnet-5-5, 300 sessions (control 150, stacklit-quiet 150), 3 repos, 10 tasks each (the round 2 tasks), 5 trials. Notional spend 5.95 USD of the 15 USD cap. No harness failures, rate limits or stopped runs. Bootstrap seed 20260508, 10,000 resamples, paired by task and trial.
+
+## Verdict
+
+Quiet mode does not pass. Q1 fails and Q3 fails on theauth. Q2 passes. No claim of "no regression below 5,000 files" is supported, and the default quiet limit is not shown to be right.
+
+| Id | Criterion | Result | Outcome |
+|---|---|---|---|
+| Q1 | Pooled cost ratio, upper bound below 1.05 | 1.08 [1.05, 1.11] | fail |
+| Q2 | Correctness difference, lower bound above -5 points | 100% in both arms (150/150) | pass |
+| Q3 | Per repo cost ratio, upper bound below 1.15 | theauth-go 1.04 [1.00, 1.09], featuredrop 1.07 [1.02, 1.11], theauth 1.13 [1.09, 1.18] | fail (theauth, 1.18) |
+
+## Other metrics, quiet against control (ratio, pooled, 150 pairs)
+
+| Metric | control | stacklit-quiet | ratio [95% CI] |
+|---|---|---|---|
+| cost (USD) | 0.0191 | 0.0206 | 1.08 [1.05, 1.11] |
+| total tokens | 16,925 | 21,710 | 1.28 [1.23, 1.34] |
+| uncached-equivalent tokens | 5,252 | 5,974 | 1.14 [1.11, 1.17] |
+| turns | 3.19 | 3.09 | 0.97 [0.92, 1.01] |
+| tool calls | 2.19 | 2.09 | 0.95 [0.89, 1.02] |
+
+The agent ran zero stacklit commands in 150 of 150 treatment sessions, and the session hook injected nothing. So the extra tokens do not come from stacklit output or from the hook text. They come from what the arm still carries: the one line rules block that `setup` writes, and the Bash tool the arm is given (control has Read, Grep and Glob only). This round cannot tell which of the two causes the cost. That is a hypothesis, not a finding.
+
+## Compared with round 2 (stacklit-e against control, different sessions)
+
+theauth-go 1.05 to 1.04, featuredrop 1.21 to 1.07, theauth 1.42 to 1.13. Quiet mode removed most of the extra cost but not all of it. These are separate runs, so the comparison is indicative only.
+
+## Deviations and limits
+
+- The analyzer did not know the arm name `stacklit-quiet`, so its first pass wrote empty tables. `treatmentArms` in `analyze.go` was extended and the analysis re-run on the same ledger. No data changed.
+- The tasks are the round 2 tasks, run again. featuredrop was contaminated in round 2's memorisation check (3 of 6 right with tools off). Without featuredrop the pooled ratio is not computed here.
+- Correctness is at ceiling (100% in both arms), so Q2 cannot detect a small loss.
+- One model, one machine, tasks written by us on repos we own, notional cost.
+
+## What this means for the default
+
+Leave quiet mode in, since it cut the cost, but do not describe it as free. The next step is a diagnostic arm that separates the two causes: control plus the Bash tool with no stacklit setup, and quiet with no rules block. It needs its own pre-registration and approval.

bench/agent-ab/analyze.go

@@ -151,7 +151,7 @@ func loadTasks(dir, repoName string) map[string]task {
 }
 
 // treatmentArms lists every arm that is compared with its base.
-var treatmentArms = []string{"stacklit", "strict", "stacklit-e", "stacklit-e-small"}
+var treatmentArms = []string{"stacklit", "strict", "stacklit-e", "stacklit-e-small", "stacklit-quiet"}
 
 func pairs(c config, repoName, arm string) []pair { return pairsAgainst(c, repoName, arm, "control") }
 

bench/agent-ab/results-round3/featuredrop/control/fd2-01-t1.json

@@ -0,0 +1,39 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-01",
+  "kind": "locate",
+  "trial": 1,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/rss.ts",
+  "turns": 4,
+  "cost_usd": 0.0320306,
+  "usage": {
+    "input_tokens": 6,
+    "output_tokens": 467,
+    "cache_creation_input_tokens": 6397,
+    "cache_read_input_tokens": 17606
+  },
+  "tool_calls": {
+    "Grep": 3
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 6.137060583,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=rss|RSS|\u003crss|atom",
+    "Grep pattern=manifest",
+    "Grep pattern=rss path=src"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:16:08Z"
+}

bench/agent-ab/results-round3/featuredrop/control/fd2-01-t2.json

@@ -0,0 +1,38 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-01",
+  "kind": "locate",
+  "trial": 2,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/rss.ts",
+  "turns": 3,
+  "cost_usd": 0.0255315,
+  "usage": {
+    "input_tokens": 4,
+    "output_tokens": 273,
+    "cache_creation_input_tokens": 5478,
+    "cache_read_input_tokens": 8815
+  },
+  "tool_calls": {
+    "Grep": 2
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 4.09440925,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=rss|RSS",
+    "Grep pattern=manifest"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:16:56Z"
+}

bench/agent-ab/results-round3/featuredrop/control/fd2-01-t3.json

@@ -0,0 +1,39 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-01",
+  "kind": "locate",
+  "trial": 3,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/rss.ts, src/bridges.ts",
+  "turns": 4,
+  "cost_usd": 0.030179599999999997,
+  "usage": {
+    "input_tokens": 6,
+    "output_tokens": 504,
+    "cache_creation_input_tokens": 5844,
+    "cache_read_input_tokens": 17516
+  },
+  "tool_calls": {
+    "Grep": 3
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 6.545461625,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=rss|RSS",
+    "Grep pattern=manifest",
+    "Grep pattern=rss path=src/bridges.ts"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:19:14Z"
+}

bench/agent-ab/results-round3/featuredrop/control/fd2-01-t4.json

@@ -0,0 +1,39 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-01",
+  "kind": "locate",
+  "trial": 4,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/rss.ts",
+  "turns": 4,
+  "cost_usd": 0.020183399999999997,
+  "usage": {
+    "input_tokens": 6,
+    "output_tokens": 437,
+    "cache_creation_input_tokens": 3572,
+    "cache_read_input_tokens": 15134
+  },
+  "tool_calls": {
+    "Grep": 3
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 5.810286583,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=rss|\u003cfeed|atom",
+    "Grep pattern=rss path=src/index.ts",
+    "Grep pattern=rss path=src/rss.ts"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:25:34Z"
+}

bench/agent-ab/results-round3/featuredrop/control/fd2-01-t5.json

@@ -0,0 +1,40 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-01",
+  "kind": "locate",
+  "trial": 5,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/rss.ts",
+  "turns": 5,
+  "cost_usd": 0.0232428,
+  "usage": {
+    "input_tokens": 8,
+    "output_tokens": 629,
+    "cache_creation_input_tokens": 3708,
+    "cache_read_input_tokens": 21048
+  },
+  "tool_calls": {
+    "Grep": 4
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 6.970601667,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=rss|RSS",
+    "Grep pattern=manifest path=/private/tmp/abreal/work/ab-1691852961/co-2542119967/src",
+    "Grep pattern=rss|manifest path=/private/tmp/abreal/work/ab-1691852961/co-2542119967/src/rss.ts",
+    "Grep pattern=rss path=/private/tmp/abreal/work/ab-1691852961/co-2542119967/src/bridges.ts"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:18:43Z"
+}

bench/agent-ab/results-round3/featuredrop/control/fd2-02-t1.json

@@ -0,0 +1,39 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-02",
+  "kind": "locate",
+  "trial": 1,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/dependencies.ts",
+  "turns": 4,
+  "cost_usd": 0.0215878,
+  "usage": {
+    "input_tokens": 8,
+    "output_tokens": 493,
+    "cache_creation_input_tokens": 3625,
+    "cache_read_input_tokens": 21418
+  },
+  "tool_calls": {
+    "Grep": 3
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 6.731618458,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=topological|toposort|cycle|circular",
+    "Grep pattern=topological|toposort|cycle path=src",
+    "Grep pattern=topological|toposort|cycle path=src"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:15:45Z"
+}

bench/agent-ab/results-round3/featuredrop/control/fd2-02-t2.json

@@ -0,0 +1,39 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-02",
+  "kind": "locate",
+  "trial": 2,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/dependencies.ts",
+  "turns": 4,
+  "cost_usd": 0.0215878,
+  "usage": {
+    "input_tokens": 8,
+    "output_tokens": 493,
+    "cache_creation_input_tokens": 3625,
+    "cache_read_input_tokens": 21418
+  },
+  "tool_calls": {
+    "Grep": 3
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 6.452275167,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=topological|toposort|cycle|circular",
+    "Grep pattern=topological|toposort|cycle path=src",
+    "Grep pattern=topological|toposort|cycle path=src"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:19:59Z"
+}

bench/agent-ab/results-round3/featuredrop/control/fd2-02-t3.json

@@ -0,0 +1,39 @@
+{
+  "repo": "featuredrop",
+  "task": "fd2-02",
+  "kind": "locate",
+  "trial": 3,
+  "arm": "control",
+  "model": "claude-sonnet-5-5",
+  "correct": true,
+  "recall": -1,
+  "answer": "src/dependencies.ts",
+  "turns": 4,
+  "cost_usd": 0.021378099999999997,
+  "usage": {
+    "input_tokens": 8,
+    "output_tokens": 477,
+    "cache_creation_input_tokens": 3613,
+    "cache_read_input_tokens": 21401
+  },
+  "tool_calls": {
+    "Grep": 3
+  },
+  "files_read": 0,
+  "read_bytes": 0,
+  "stacklit_calls": 0,
+  "hook_events": 0,
+  "crawl_guard_fired": false,
+  "crawl_redirects": 0,
+  "permission_denials": 0,
+  "result_subtype": "success",
+  "seconds": 12.968828167,
+  "seven_day_utilization": 0.69,
+  "call_log": [
+    "Grep pattern=topological|toposort|cycle|circular",
+    "Grep pattern=topological|toposort|cycle path=src",
+    "Grep pattern=topological|toposort|cycle path=src"
+  ],
+  "stacklit_bin": "0870521fa963",
+  "at": "2026-11-09T04:24:25Z"
+}