Release v2.0.0: bench link

Author: ProjectBarks

writing/headroom-fewer-tokens-bigger-bill.html

@@ -2,7 +2,7 @@
 <p>i haven’t seen much a/b testing out of any of them, and the optimizations don’t make much sense to me in an efficient market. i don’t have a good name for this, so i’ll call it the platform envelopment paradox: why wouldn’t anthropic or openai adopt these tricks internally and forward a chunk of the savings to you? headroom, rtk and ponytail all save tokens, and i’d imagine most engineers at those labs spend their days looking for simple ways to save compute. any major token optimization that can be lifted onto the platform probably will be. so i wanted to actually evaluate one in a controlled way. funny enough, jetbrains had already <a href="https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/">tested</a> rtk and found it increased costs, so i ran the same kind of test on headroom.</p>
 <h2 id="the-plan">the plan</h2>
 <p>most of headroom’s <a href="https://headroom-docs.vercel.app/docs/benchmarks">benchmarks</a> compare headroom to itself. run it locally and you get numbers telling you how much money you’re saving, with no way to say how the same non-deterministic entropy hole would have behaved without it.</p>
-<p>so i used <a href="https://www.harborframework.com/">harbor</a> with the sr swe bench, three arms over the same tasks: plain claude as the baseline, headroom in token mode, and headroom in cache mode<sup><a href="#user-content-fn-1" id="user-content-fnref-1" data-footnote-ref="true" aria-describedby="footnote-label">1</a></sup>. cache mode changes so little that it mostly measures how far two runs of the same setup drift apart on their own, and that number ends up doing a lot of work later. running all of it meant spinning up a pile of vms on an anthropic api key (no subscription pricing), so i sampled 25 test cases picked to favor headroom.</p>
+<p>so i used <a href="https://www.harborframework.com/">harbor</a> with the <a href="https://hub.harborframework.com/datasets/snorkel-ai/senior-swe-bench-v2026.06">sr swe bench</a>, three arms over the same tasks: plain claude as the baseline, headroom in token mode, and headroom in cache mode<sup><a href="#user-content-fn-1" id="user-content-fnref-1" data-footnote-ref="true" aria-describedby="footnote-label">1</a></sup>. cache mode changes so little that it mostly measures how far two runs of the same setup drift apart on their own, and that number ends up doing a lot of work later. running all of it meant spinning up a pile of vms on an anthropic api key (no subscription pricing), so i sampled 25 test cases picked to favor headroom.</p>
 <h2 id="what-i-found">what i found</h2>
 <p>token mode<sup><a href="#user-content-fn-1" id="user-content-fnref-1-2" data-footnote-ref="true" aria-describedby="footnote-label">1</a></sup> cut total input tokens from 215m to 132m across the arm. that part works exactly as advertised. <em>yay</em></p>
 <figure class="fig"><div class="fig-pair"><div><p class="fig-head">total prompt tokens</p><p class="fig-sub">cache reads plus fresh input</p><div class="plot-wide"><svg class="plot-d6a7b5" fill="currentColor" font-family="system-ui, sans-serif" font-size="10" text-anchor="middle" height="168" viewBox="0 0 337 168" style="background: transparent; font-size: 11px;" role="img" aria-label="total prompt tokens: baseline 215M, cache mode 191M, token mode 132M."><style>:where(.plot-d6a7b5) {