From 9290052f376f5d4d206e4d640466e3a8bb1199be Mon Sep 17 00:00:00 2001 From: bompus Date: Sun, 1 Nov 2026 11:24:05 -0700 Subject: [PATCH 1/8] docs: re-measure the fork against upstream on 2026-11-01 Full seven-corpus Node comparison at fork 8313e9a0 against upstream cd04b21a, timed end to end with Hyperfine 2.0.0. The runner change means every arm was re-run; the 2026-10-31 figures are not mixed in. The fork indexes faster on five of seven (1% pretix to 1.79x CPython); gin and supabase are inside their spreads. Supabase is no longer an 11% loss: 1.9% inside a 2.1% fork spread on 18.5% more nodes. Sync is faster on six of seven (1.22x to 2.63x); discourse is 3% slower. README chart and tables updated, new benchmark record added; MCP left at its 2026-10-31 measurement and labelled as not re-run. --- README.md | 56 +++++++-------- assets/fork-vs-upstream-performance.svg | 68 +++++++++---------- .../benchmarks/fork-vs-upstream-2026-11-01.md | 48 +++++++++++++ 3 files changed, 110 insertions(+), 62 deletions(-) create mode 100644 docs/benchmarks/fork-vs-upstream-2026-11-01.md diff --git a/README.md b/README.md index c96121653..ee7b19bd8 100644 --- a/README.md +++ b/README.md @@ -198,51 +198,51 @@ Call resolution the fork adds: ### Measured results -Upstream `main` at `cd04b21a` against the fork at `c63e3d7c`, Node.js 24.22.2 on the same 24-core bare-metal Linux host, measured 2026-10-31 under the host benchmark lock. This is the second 2026-10-31 run, after the prerequisite-pool fix below; the first one exposed that regression and is kept as [its own record](docs/benchmarks/fork-vs-upstream-2026-10-31.md). Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked no clear change rather than given a fixed band. Bun was not re-measured in this run; the Bun figures below are from the 2026-10-28 campaign, and the CLI table is from the 2026-10-27 run at `bc27fdb4`. The numbers, method, spreads, graph sizes and admission receipts are in [`docs/benchmarks/fork-vs-upstream-2026-10-31b.md`](docs/benchmarks/fork-vs-upstream-2026-10-31b.md). The scripts are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). +Upstream `main` at `cd04b21a` against the fork at `8313e9a0`, Node.js 24.22.2 on the same 24-core bare-metal Linux host, measured 2026-11-01 under the host benchmark lock with Hyperfine 2.0.0. Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked no clear change rather than given a fixed band. Bun was not re-measured in this run; the Bun figures below are from the 2026-10-28 campaign, and the CLI table is from the 2026-10-27 run at `bc27fdb4`. The numbers, method, spreads, graph sizes and admission receipts are in [`docs/benchmarks/fork-vs-upstream-2026-11-01.md`](docs/benchmarks/fork-vs-upstream-2026-11-01.md). The scripts are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). -![Fork against upstream CodeGraph on seven corpora, Node, 2026-10-31: the fork syncs a one-file edit 1.08 to 2.69 times faster, indexes CPython 1.71 times faster and n8n 1.32 times faster, and trails gin by 8% and supabase by 11% on a graph with 33% more nodes.](assets/fork-vs-upstream-performance.svg?v=13) +![Fork against upstream CodeGraph on seven corpora, Node, 2026-11-01: the fork syncs six of seven one-file edits 1.22 to 2.63 times faster (discourse 3% slower), indexes CPython 1.79 times faster and n8n 1.43 times faster, and is level with gin and supabase, the latter on 18.5% more nodes.](assets/fork-vs-upstream-performance.svg?v=14) -*2026-10-31 Node run, fork `c63e3d7c` against upstream `cd04b21a`. Ratios are exact; the tables mark which gaps clear the run-to-run spread.* +*2026-11-01 Node run, fork `8313e9a0` against upstream `cd04b21a`. Ratios are exact; the tables mark which gaps clear the run-to-run spread.* -The fork's one-file sync is faster on all seven corpora: 8% (discourse) to 63% (n8n), or 1.08× to 2.69×. Full indexing: CPython is 1.71× faster (27.06 s against 46.34 s), n8n 1.32× faster (53.95 s against 71.36 s), pretix 11% faster and Alamofire and discourse 4–5% faster; gin is 8% slower (0.07 s absolute on a 119-file repo) and supabase 11% slower on a graph with 33% more nodes and 12% more edges. The speed costs memory on the smaller corpora — index peaks +50% on gin, +37% on Alamofire, +23% on pretix and +1% on supabase — while it is lower on CPython (−17%), discourse (−12%) and n8n (−27%). MCP on n8n: warm explore 1.75× faster, the eight-explore wave 1.26× faster and `initialize` 1.79× faster, with busy memory 21% lower and idle memory 51% lower; the watcher's one-file re-index is the one loss, 9% slower (766 ms against 702 ms). +The fork's one-file sync is faster on six of seven corpora: 22% (gin) to 163% (n8n), or 1.22× to 2.63×; discourse is 3% slower. Full indexing: CPython is 1.79× faster (25.84 s against 46.36 s), n8n 1.43× faster (50.16 s against 71.75 s), discourse 8% faster, Alamofire 5% faster and pretix 1% faster; gin and supabase are no clear change. Supabase is 1.9% slower inside a 2.1% fork spread, on a graph with 18.5% more nodes and 6.7% more edges. The speed costs memory on the smaller corpora — index peaks +49% on gin, +34% on Alamofire and +16% on pretix — while it is lower on CPython (−20%), discourse (−8%), supabase (−1%) and n8n (−29%). MCP on n8n (2026-10-31, not re-run): warm explore 1.75× faster, the eight-explore wave 1.26× faster and `initialize` 1.79× faster, with busy memory 21% lower and idle memory 51% lower; the watcher's one-file re-index is the one loss, 9% slower (766 ms against 702 ms). -The [2026-10-28 Node+Bun campaign](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md), the [2026-10-30 Node re-measurement](docs/benchmarks/fork-vs-upstream-node-2026-10-30.md) and the [first 2026-10-31 run](docs/benchmarks/fork-vs-upstream-2026-10-31.md) are superseded by this run and kept with their original dates and commits. +The [2026-10-28 Node+Bun campaign](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md), the [2026-10-30 Node re-measurement](docs/benchmarks/fork-vs-upstream-node-2026-10-30.md), the [first 2026-10-31 run](docs/benchmarks/fork-vs-upstream-2026-10-31.md) and the [second 2026-10-31 run](docs/benchmarks/fork-vs-upstream-2026-10-31b.md) are superseded by this run and kept with their original dates and commits. The fork starts its resolver pool at 12,000 pending references on a full index (150,000 before #590). Below that, it can start the pool part-way through if at least 20,000 unresolved references remain and the estimated main-thread work is at least 0.4 s. An explicit `CODEGRAPH_PARALLEL_RESOLVE_MIN` replaces both gates. The pool sizes itself at about one worker per 2,000 refs, capped at 25,000 refs per worker and by CPU and memory headroom (8 workers on gin, 12 on the larger corpora here); the bulk edge/ref index windows keep their own 150,000-ref floor. Workers spawn as the pool is gated in and open a pre-prerequisite snapshot, so the prerequisite batches resolve across the pool; a second snapshot is taken once their supertype edges persist and the workers recycle onto it. Resolution proceeds in pages of 40,000 references (`CODEGRAPH_RESOLVE_BATCH` overrides), across up to 12 workers (`CODEGRAPH_RESOLVE_WORKERS` sets the count, `0` disables the pool; `CODEGRAPH_NO_PARALLEL_RESOLVE=1` does the same). -**Full index** (`codegraph init`), Node, 2026-10-31; time, peak memory and graph: +**Full index** (`codegraph init`), Node, 2026-11-01; time, peak memory and graph: | Corpus | Upstream | Fork | Fork / upstream | Upstream nodes / edges | Fork nodes / edges | |---|---:|---:|---:|---:|---:| -| gin (Go, 119 files) | 0.81 s, 496 MiB | 0.88 s, 744 MiB | **8% slower** | 2,660 / 8,050 | 3,089 / 8,546 | -| Alamofire (Swift, 129 files) | 1.23 s, 850 MiB | 1.17 s, 1,168 MiB | **4% faster** | 4,600 / 12,512 | 5,614 / 13,682 | -| pretix (Python + JS, 1,472 files) | 9.13 s, 3,547 MiB | 8.08 s, 4,367 MiB | **11% faster** | 32,783 / 79,319 | 32,799 / 73,769 | -| CPython (C + Python, 3,708 files) | 46.34 s, 7,388 MiB | 27.06 s, 6,168 MiB | **1.71× faster** | 169,618 / 615,814 | 170,311 / 605,273 | -| discourse (Ruby + JS, 20,277 files) | 15.13 s, 4,440 MiB | 14.38 s, 3,906 MiB | **5% faster** | 163,411 / 314,691 | 168,110 / 310,348 | -| supabase (React + Next.js + TS, 10,716 files) | 15.70 s, 5,830 MiB | 17.37 s, 5,877 MiB | **11% slower** | 114,440 / 292,021 | 152,274 / 328,137 | -| n8n (Vue + TS, 24,434 files) | 71.36 s, 12,219 MiB | 53.95 s, 8,867 MiB | **1.32× faster** | 315,422 / 1,239,521 | 328,519 / 1,236,342 | +| gin (Go, 119 files) | 0.90 s, 484 MiB | 0.89 s, 721 MiB | **no clear change** | 2,660 / 8,050 | 2,985 / 8,442 | +| Alamofire (Swift, 129 files) | 1.24 s, 844 MiB | 1.18 s, 1,133 MiB | **5% faster** | 4,600 / 12,512 | 5,587 / 13,655 | +| pretix (Python + JS, 1,472 files) | 9.20 s, 3,574 MiB | 9.08 s, 4,129 MiB | **1% faster** | 32,781 / 79,317 | 32,795 / 73,765 | +| CPython (C + Python, 3,708 files) | 46.36 s, 7,588 MiB | 25.84 s, 6,051 MiB | **1.79× faster** | 169,618 / 615,814 | 169,698 / 604,660 | +| discourse (Ruby + JS, 20,277 files) | 15.33 s, 4,398 MiB | 14.13 s, 4,064 MiB | **8% faster** | 163,411 / 314,691 | 166,865 / 309,103 | +| supabase (React + Next.js + TS, 10,716 files) | 15.73 s, 5,891 MiB | 16.03 s, 5,822 MiB | **no clear change** | 114,440 / 292,021 | 135,658 / 311,496 | +| n8n (Vue + TS, 24,434 files) | 71.75 s, 12,252 MiB | 50.16 s, 8,754 MiB | **1.43× faster** | 315,422 / 1,239,521 | 324,381 / 1,232,201 | -The fork indexes faster on five of seven corpora, by 4% (Alamofire) to 1.71× (CPython). pretix's 11% is close to its 9% fork spread but clears it. The two losses are gin (8%, 0.07 s absolute on a 119-file repo) and supabase (11% on a graph with 33% more nodes and 12% more edges; per node the fork is faster). The fork's larger graphs are part of the gin, Alamofire and supabase difference; pretix and CPython are within 1% on nodes. Peak-memory change, fork against upstream: gin +50%, Alamofire +37%, pretix +23%, CPython −17%, discourse −12%, supabase +1%, n8n −27%. +The fork indexes faster on five of seven corpora, by 1% (pretix) to 1.79× (CPython). Gin and supabase are no clear change: gin's 0.7% gap is inside its 5.7% spread, and supabase's 1.9% is inside its 2.1% fork spread on a graph with 18.5% more nodes and 6.7% more edges. Peak-memory change, fork against upstream: gin +49%, Alamofire +34%, pretix +16%, CPython −20%, discourse −8%, supabase −1%, n8n −29%. **The 2026-10-31 regression is closed.** The first run that day showed supabase 36% and n8n level, but the fork's own index had slowed at matched graph sizes. A bisect named #591 (`05077514`): it moved the prerequisite (imports/extends/implements) batches to the main-thread kernel so the workers could open one post-prerequisite snapshot, serializing that phase on one core — supabase 17.27/17.46 s to 21.84/22.51 s, n8n 54.46/57.22 s to 70.34/70.70 s, index CPU ~630% to ~460%. #590 alone was inside the pre-#590 spread. `c63e3d7c` opens the pool on the pre-prerequisite snapshot again and recycles it onto a second snapshot after the supertypes persist; the interleaved A/B at fixed graph sizes was supabase 21.60/21.88 s → 17.11/17.27 s and n8n 70.72/73.17 s → 53.58/53.91 s. The cost is peak memory on n8n: 8.8–8.9 GiB against 8.2–8.5 GiB, still below upstream's 11.9 GiB. Evidence: [the fix's measurement doc](docs/benchmarks/fork-vs-upstream-2026-10-31b.md) and the [pre-fix run](docs/benchmarks/fork-vs-upstream-2026-10-31.md). -**One-file sync** (edit one file, `codegraph sync`), Node, 2026-10-31: +**One-file sync** (edit one file, `codegraph sync`), Node, 2026-11-01: | Corpus | Upstream | Fork | Upstream / fork | |---|---:|---:|---:| -| gin | 0.37 s | 0.30 s | **1.23×** | -| Alamofire | 0.39 s | 0.28 s | **1.39×** | -| pretix | 1.50 s | 0.93 s | **1.62×** | -| CPython | 5.21 s | 2.54 s | **2.05×** | -| discourse | 1.06 s | 0.98 s | **1.08×** | -| supabase | 7.49 s | 3.45 s | **2.17×** | -| n8n | 11.37 s | 4.23 s | **2.69×** | +| gin | 0.395 s | 0.325 s | **1.22×** | +| Alamofire | 0.412 s | 0.295 s | **1.40×** | +| pretix | 1.513 s | 0.901 s | **1.68×** | +| CPython | 5.231 s | 2.596 s | **2.02×** | +| discourse | 1.070 s | 1.101 s | **0.97× (3% slower)** | +| supabase | 7.585 s | 3.450 s | **2.20×** | +| n8n | 11.481 s | 4.368 s | **2.63×** | -Every cell clears its run-to-run spread; discourse's 8% is the smallest. +Six of seven clear their run-to-run spread and are faster; discourse is 3% slower, the one sync loss. Both builds rebuild the links inferred from events, callbacks and function pointers after a sync, so a sync ends with the graph a full index would build ([colbymchenry/codegraph#1988](https://github.com/colbymchenry/codegraph/issues/1988)). Upstream reruns every inference pass inside each sync that touches an inferred link; the fork reruns the same passes a full index runs once edits pause. `CODEGRAPH_SYNC_RESYNTHESIS=0` turns the fork's rebuild off, trading it for stale inferred links. -**MCP server** on n8n (`codegraph serve --mcp`; memory and CPU summed over the server and the daemon), Node, 2026-10-31: +**MCP server** on n8n (`codegraph serve --mcp`; memory and CPU summed over the server and the daemon), Node, 2026-10-31 (not re-run on 2026-11-01; its process-tree harness is outside the Hyperfine campaign and the Markdown work changes index nodes, not the query path): | | Upstream | Fork | |---|---:|---:| @@ -256,9 +256,9 @@ Both builds rebuild the links inferred from events, callbacks and function point | Memory at idle | 4.46 GiB | 2.19 GiB | | CPU at idle, share of one core | 0.40% | 0.20% | -The idle row is the process tree's RSS at the harness's idle sample, 70 s after the last query. The same measure across runs: **3.58 GiB** on 2026-10-27 (`bc27fdb4`, the old README row), **4.21 GiB** on 2026-10-28 (`5784e289`, the clean re-measure), **19.7 GiB** on 2026-10-30 (`a9dc5a61`, a real regression), **2.13 GiB** in the first 2026-10-31 run (`cf2a37a5`, #593) and **2.19 GiB** now (`c63e3d7c`). The “20 GB → 2 GB” statement is the 2026-10-30 to 2026-10-31 change on this harness, not a different definition from the earlier rows. +The idle row is the process tree's RSS at the harness's idle sample, 70 s after the last query. The same measure across runs: **3.58 GiB** on 2026-10-27 (`bc27fdb4`, the old README row), **4.21 GiB** on 2026-10-28 (`5784e289`, the clean re-measure), **19.7 GiB** on 2026-10-30 (`a9dc5a61`, a real regression), **2.13 GiB** in the first 2026-10-31 run (`cf2a37a5`, #593) and **2.19 GiB** at the 2026-10-31 run (`c63e3d7c`). The “20 GB → 2 GB” statement is the 2026-10-30 to 2026-10-31 change on this harness, not a different definition from the earlier rows. -**CLI startup** on gin (`hyperfine`, mean of 20 runs), 2026-10-27, fork `bc27fdb4` against upstream `69af19ac`; not re-measured on 2026-10-31: +**CLI startup** on gin (`hyperfine`, mean of 20 runs), 2026-10-27, fork `bc27fdb4` against upstream `69af19ac`; not re-measured on 2026-11-01: | Command | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| @@ -281,9 +281,9 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne ### What it costs -- **Larger database:** 12–20% bigger on six of the seven corpora above, and 38% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. +- **Larger database:** 11–20% bigger on six of the seven corpora above, and 31.5% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. - **Fewer edges on some projects:** 7% fewer on pretix and 1.7% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. -- **More memory while indexing on four projects:** in the 2026-10-31 comparison the fork's full index peaks higher on gin (+50%), Alamofire (+37%), pretix (+23%) and supabase (+1%) — the in-process resolver-pool workers used on small and mid-size projects, plus the second pool snapshot on large ones — while it is lower on CPython (−17%), discourse (−12%) and n8n (−27%). +- **More memory while indexing on the smaller projects:** in the 2026-11-01 comparison the fork's full index peaks higher on gin (+49%), Alamofire (+34%) and pretix (+16%) — the in-process resolver-pool workers used on small and mid-size projects — while it is lower on CPython (−20%), discourse (−8%), supabase (−1%) and n8n (−29%). - **MCP watcher sync:** the daemon's one-file re-index is 9% slower (766 ms against 702 ms) because it also runs the deferred synthesized-edge refresh; the CLI one-file sync is faster on all seven corpora. --- diff --git a/assets/fork-vs-upstream-performance.svg b/assets/fork-vs-upstream-performance.svg index ec1d74ec3..c301cbfcc 100644 --- a/assets/fork-vs-upstream-performance.svg +++ b/assets/fork-vs-upstream-performance.svg @@ -1,10 +1,10 @@ - -Fork versus upstream on seven corpora, Node, 2026-10-31 + +Fork versus upstream on seven corpora, Node, 2026-11-01 Two panels: full index fork/upstream ratios and one-file sync upstream/fork speedups for gin, Alamofire, pretix, CPython, discourse, supabase and n8n. -Fork vs upstream on Node — seven corpora, 2026-10-31 -Upstream cd04b21a · fork c63e3d7c · Node 24.22.2 · Node-only run +Fork vs upstream on Node — seven corpora, 2026-11-01 +Upstream cd04b21a · fork 8313e9a0 · Node 24.22.2 · Hyperfine 2.0.0 Index: median of 2 runs, fork / upstream (lower is better). Sync: median of 4 runs, upstream / fork (higher is better). Full index @@ -18,26 +18,26 @@ 1.5× gin (Go) - -1.08× + +0.99× Alamofire (Swift) - -0.96× + +0.95× pretix (Python) - -0.89× + +0.99× CPython (C/Py) - -0.58× + +0.56× discourse (Ruby) - -0.95× + +0.92× supabase (TS) - -1.11× + +1.02× n8n (Vue/TS) - -0.76× + +0.70× One-file sync 0.5× @@ -52,27 +52,27 @@ 3× gin (Go) - -1.23× + +1.22× Alamofire (Swift) - -1.39× + +1.40× pretix (Python) - -1.62× + +1.68× CPython (C/Py) - -2.05× + +2.02× discourse (Ruby) - -1.08× + +0.97× supabase (TS) - -2.17× + +2.20× n8n (Vue/TS) - -2.69× + +2.63× Exact ratios; the tables mark which gaps clear the larger arm's run-to-run spread. -Method, spreads, graph sizes, admission receipts: docs/benchmarks/fork-vs-upstream-2026-10-31b.md +Method, spreads, graph sizes, admission receipts: docs/benchmarks/fork-vs-upstream-2026-11-01.md CodeGraph fork · bompus/codegraph · measured under the host benchmark lock - \ No newline at end of file + diff --git a/docs/benchmarks/fork-vs-upstream-2026-11-01.md b/docs/benchmarks/fork-vs-upstream-2026-11-01.md new file mode 100644 index 000000000..415e5845e --- /dev/null +++ b/docs/benchmarks/fork-vs-upstream-2026-11-01.md @@ -0,0 +1,48 @@ +# Fork against upstream on Node, 2026-11-01, after the Markdown node-model work + +A Node-only comparison at the fork head that lands the Markdown node-model work (#612 and #613). It supersedes [fork-vs-upstream-2026-10-31b.md](fork-vs-upstream-2026-10-31b.md) for every figure it covers. The runner changed from the hand-written `/usr/bin/time` loop to Hyperfine 2.0.0, so every arm was re-run here; figures are not directly comparable to the earlier run. + +- Upstream: `cd04b21a` (`fix(python): a base's type argument is no supertype…`). +- Fork: `8313e9a0` (`fix(markdown): fold commands and table rows into their section (#613)`). +- Host: 24-core bare-metal Ubuntu, 122 GiB, Node.js 24.22.2, release kernel. +- Corpora: gin, Alamofire, pretix, CPython, discourse, supabase, n8n. +- Runner: Hyperfine 2.0.0 for whole commands. Index is one Hyperfine run per invocation and two invocations per arm (median of 2); one-file sync is two Hyperfine runs per invocation and two invocations per arm (median of 4). Arms run in mirrored order (upstream, fork, fork, upstream). +- Peak memory is Hyperfine's `memory_peak_resident` maximum. Graph counts are read from the fresh index after its timed run. +- A difference counts only when the fork/upstream gap exceeds the larger of the two arms' `(max − min) / median` for that metric. Spreads are listed per cell. +- The host benchmark lock was held for each corpus and a fresh 10-second admission check passed before it started. Receipts: `~/cg-scratch/bench-2026-11-01/admission-.json`; raw Hyperfine JSON and `graph.tsv` beside them. + +## Full index + +| Corpus | Upstream | Fork | Fork / upstream | Verdict | Upstream RSS | Fork RSS | Nodes up/fork | Edges up/fork | +|---|---:|---:|---:|---|---:|---:|---:|---:| +| gin | 0.899 s | 0.893 s | 0.993 | no clear change (spreads 5.3% / 5.7%) | 484 MiB | 721 MiB | 2,660 / 2,985 | 8,050 / 8,442 | +| Alamofire | 1.238 s | 1.175 s | 0.949 | **5% faster** (0.7% / 0.7%) | 844 MiB | 1,133 MiB | 4,600 / 5,587 | 12,512 / 13,655 | +| pretix | 9.198 s | 9.079 s | 0.987 | **1% faster** (0.2% / 0.7%) | 3,574 MiB | 4,129 MiB | 32,781 / 32,795 | 79,317 / 73,765 | +| CPython | 46.357 s | 25.842 s | 0.557 | **1.79× faster** (0.0% / 0.1%) | 7,588 MiB | 6,051 MiB | 169,618 / 169,698 | 615,814 / 604,660 | +| discourse | 15.329 s | 14.128 s | 0.922 | **8% faster** (0.4% / 0.7%) | 4,398 MiB | 4,064 MiB | 163,411 / 166,865 | 314,691 / 309,103 | +| supabase | 15.732 s | 16.028 s | 1.019 | no clear change (0.4% / 2.1%), 18.5% more nodes | 5,891 MiB | 5,822 MiB | 114,440 / 135,658 | 292,021 / 311,496 | +| n8n | 71.748 s | 50.159 s | 0.699 | **1.43× faster** (0.4% / 0.4%) | 12,252 MiB | 8,754 MiB | 315,422 / 324,381 | 1,239,521 / 1,232,201 | + +The fork indexes faster on five of seven corpora, by 1% (pretix) to 1.79× (CPython). Gin and supabase are no clear change: gin's 0.7% gap is far inside its 5.7% spread, and supabase's 1.9% is inside its 2.1% fork spread. Supabase is no longer an 11% loss; its graph is 18.5% larger in nodes and 6.7% larger in edges than upstream's. Peak-memory change, fork against upstream: gin +49%, Alamofire +34%, pretix +16%, CPython −20%, discourse −8%, supabase −1%, n8n −29%. + +The supabase node count fell from 152,274 to 135,658 across #612 and #613; the import-node and command/table-row work removed 16,616 nodes and 16,641 edges. CPython's fork graph stays within 0.1% of upstream's node count. + +## One-file sync + +| Corpus | Upstream | Fork | Upstream / fork | Verdict | +|---|---:|---:|---:|---| +| gin | 0.395 s | 0.325 s | 1.22× | **18% faster** (2.7% / 3.4%) | +| Alamofire | 0.412 s | 0.295 s | 1.40× | **28% faster** (0.4% / 2.9%) | +| pretix | 1.513 s | 0.901 s | 1.68× | **40% faster** (1.3% / 2.7%) | +| CPython | 5.231 s | 2.596 s | 2.02× | **50% faster** (0.5% / 1.1%) | +| discourse | 1.070 s | 1.101 s | 0.97× | **3% slower** (1.8% / 1.5%) | +| supabase | 7.585 s | 3.450 s | 2.20× | **55% faster** (2.3% / 0.8%) | +| n8n | 11.481 s | 4.368 s | 2.63× | **62% faster** (0.4% / 1.4%) | + +The fork syncs faster on six of seven corpora. Discourse is 3% slower, the one sync loss; its 2.9% gap clears the larger arm spread (1.8%). The other six clear their spreads too. + +## Notes + +- **Runner change.** This is the first campaign timed with Hyperfine 2.0.0 end to end. Every arm was re-run; the 2026-10-31 figures are not directly comparable and should not be mixed with these. +- **MCP not re-run.** The MCP table in the README is still the 2026-10-31 measurement (its harness is a process-tree observer, not one of the three required runners). The Markdown node-model work changes index-time nodes, not the MCP query path, so it is left dated and unchanged until the MCP harness is re-admitted. +- **Raw evidence.** `~/cg-scratch/bench-2026-11-01/`: `*-index.json` / `*-sync.json`, `graph.tsv`, `*-index.txt` / `*-sync.txt`, and `admission-*.json`. From 88f9364a40b0ab7d07529df8a1332a278d6ed6ab Mon Sep 17 00:00:00 2001 From: bompus Date: Sun, 1 Nov 2026 11:51:35 -0700 Subject: [PATCH 2/8] docs: re-run the n8n MCP table for 2026-11-01 The MCP table used the existing process-tree harness (mcp-bench.mjs), not Hyperfine: warm explore 142 ms against 244 ms, eight explores 1.89 s against 2.40 s, initialize 62 ms against 120 ms, idle memory 2.08 GiB against 4.31 GiB. The watcher sync is 12% slower (754 ms against 676 ms). The table stays separate from the Hyperfine index/sync figures. --- README.md | 26 +++++++++---------- .../benchmarks/fork-vs-upstream-2026-11-01.md | 2 +- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index ee7b19bd8..9107410cc 100644 --- a/README.md +++ b/README.md @@ -204,7 +204,7 @@ Upstream `main` at `cd04b21a` against the fork at `8313e9a0`, Node.js 24.22.2 on *2026-11-01 Node run, fork `8313e9a0` against upstream `cd04b21a`. Ratios are exact; the tables mark which gaps clear the run-to-run spread.* -The fork's one-file sync is faster on six of seven corpora: 22% (gin) to 163% (n8n), or 1.22× to 2.63×; discourse is 3% slower. Full indexing: CPython is 1.79× faster (25.84 s against 46.36 s), n8n 1.43× faster (50.16 s against 71.75 s), discourse 8% faster, Alamofire 5% faster and pretix 1% faster; gin and supabase are no clear change. Supabase is 1.9% slower inside a 2.1% fork spread, on a graph with 18.5% more nodes and 6.7% more edges. The speed costs memory on the smaller corpora — index peaks +49% on gin, +34% on Alamofire and +16% on pretix — while it is lower on CPython (−20%), discourse (−8%), supabase (−1%) and n8n (−29%). MCP on n8n (2026-10-31, not re-run): warm explore 1.75× faster, the eight-explore wave 1.26× faster and `initialize` 1.79× faster, with busy memory 21% lower and idle memory 51% lower; the watcher's one-file re-index is the one loss, 9% slower (766 ms against 702 ms). +The fork's one-file sync is faster on six of seven corpora: 22% (gin) to 163% (n8n), or 1.22× to 2.63×; discourse is 3% slower. Full indexing: CPython is 1.79× faster (25.84 s against 46.36 s), n8n 1.43× faster (50.16 s against 71.75 s), discourse 8% faster, Alamofire 5% faster and pretix 1% faster; gin and supabase are no clear change. Supabase is 1.9% slower inside a 2.1% fork spread, on a graph with 18.5% more nodes and 6.7% more edges. The speed costs memory on the smaller corpora — index peaks +49% on gin, +34% on Alamofire and +16% on pretix — while it is lower on CPython (−20%), discourse (−8%), supabase (−1%) and n8n (−29%). MCP on n8n: warm explore 1.72× faster, the eight-explore wave 1.27× faster and `initialize` 1.94× faster, with busy memory 19% lower and idle memory 52% lower; the watcher's one-file re-index is the one loss, 12% slower (754 ms against 676 ms). The [2026-10-28 Node+Bun campaign](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md), the [2026-10-30 Node re-measurement](docs/benchmarks/fork-vs-upstream-node-2026-10-30.md), the [first 2026-10-31 run](docs/benchmarks/fork-vs-upstream-2026-10-31.md) and the [second 2026-10-31 run](docs/benchmarks/fork-vs-upstream-2026-10-31b.md) are superseded by this run and kept with their original dates and commits. @@ -242,21 +242,21 @@ Six of seven clear their run-to-run spread and are faster; discourse is 3% slowe Both builds rebuild the links inferred from events, callbacks and function pointers after a sync, so a sync ends with the graph a full index would build ([colbymchenry/codegraph#1988](https://github.com/colbymchenry/codegraph/issues/1988)). Upstream reruns every inference pass inside each sync that touches an inferred link; the fork reruns the same passes a full index runs once edits pause. `CODEGRAPH_SYNC_RESYNTHESIS=0` turns the fork's rebuild off, trading it for stale inferred links. -**MCP server** on n8n (`codegraph serve --mcp`; memory and CPU summed over the server and the daemon), Node, 2026-10-31 (not re-run on 2026-11-01; its process-tree harness is outside the Hyperfine campaign and the Markdown work changes index nodes, not the query path): +**MCP server** on n8n (`codegraph serve --mcp`; memory and CPU summed over the server and the daemon), Node, 2026-11-01. This table uses the existing process-tree harness (`scripts/benchmarks/runtime/mcp-bench.mjs`), not Hyperfine; the required runners cannot measure per-query MCP latency, concurrent explores, process-tree memory/CPU or the watcher sync. It is a separate measurement from the Hyperfine index/sync figures: | | Upstream | Fork | |---|---:|---:| -| `initialize` answered | 115 ms | 64 ms | -| Start to first explore answer | 2.91 s | 2.70 s | -| Explore, warm median | 240 ms | 138 ms | -| Explore, p90 | 282 ms | 178 ms | -| 8 explores at once | 2.38 s | 1.88 s | -| Memory while busy | 3.81 GiB | 3.02 GiB | -| Edited file re-indexed by the watcher | 702 ms | 766 ms | -| Memory at idle | 4.46 GiB | 2.19 GiB | -| CPU at idle, share of one core | 0.40% | 0.20% | - -The idle row is the process tree's RSS at the harness's idle sample, 70 s after the last query. The same measure across runs: **3.58 GiB** on 2026-10-27 (`bc27fdb4`, the old README row), **4.21 GiB** on 2026-10-28 (`5784e289`, the clean re-measure), **19.7 GiB** on 2026-10-30 (`a9dc5a61`, a real regression), **2.13 GiB** in the first 2026-10-31 run (`cf2a37a5`, #593) and **2.19 GiB** at the 2026-10-31 run (`c63e3d7c`). The “20 GB → 2 GB” statement is the 2026-10-30 to 2026-10-31 change on this harness, not a different definition from the earlier rows. +| `initialize` answered | 120 ms | 62 ms | +| Start to first explore answer | 2.85 s | 2.75 s | +| Explore, warm median | 244 ms | 142 ms | +| Explore, p90 | 286 ms | 186 ms | +| 8 explores at once | 2.40 s | 1.89 s | +| Memory while busy | 3.66 GiB | 2.97 GiB | +| Edited file re-indexed by the watcher | 676 ms | 754 ms | +| Memory at idle | 4.31 GiB | 2.08 GiB | +| CPU at idle, share of one core | 0.38% | 0.19% | + +The idle row is the process tree's RSS at the harness's idle sample, 70 s after the last query. The same measure across runs: **3.58 GiB** on 2026-10-27 (`bc27fdb4`, the old README row), **4.21 GiB** on 2026-10-28 (`5784e289`, the clean re-measure), **19.7 GiB** on 2026-10-30 (`a9dc5a61`, a real regression), **2.13 GiB** in the first 2026-10-31 run (`cf2a37a5`, #593), **2.19 GiB** at the 2026-10-31 run (`c63e3d7c`) and **2.08 GiB** on 2026-11-01 (`8313e9a0`). The “20 GB → 2 GB” statement is the 2026-10-30 to 2026-10-31 change on this harness, not a different definition from the earlier rows. **CLI startup** on gin (`hyperfine`, mean of 20 runs), 2026-10-27, fork `bc27fdb4` against upstream `69af19ac`; not re-measured on 2026-11-01: diff --git a/docs/benchmarks/fork-vs-upstream-2026-11-01.md b/docs/benchmarks/fork-vs-upstream-2026-11-01.md index 415e5845e..7f050c4f6 100644 --- a/docs/benchmarks/fork-vs-upstream-2026-11-01.md +++ b/docs/benchmarks/fork-vs-upstream-2026-11-01.md @@ -44,5 +44,5 @@ The fork syncs faster on six of seven corpora. Discourse is 3% slower, the one s ## Notes - **Runner change.** This is the first campaign timed with Hyperfine 2.0.0 end to end. Every arm was re-run; the 2026-10-31 figures are not directly comparable and should not be mixed with these. -- **MCP not re-run.** The MCP table in the README is still the 2026-10-31 measurement (its harness is a process-tree observer, not one of the three required runners). The Markdown node-model work changes index-time nodes, not the MCP query path, so it is left dated and unchanged until the MCP harness is re-admitted. +- **MCP.** The n8n MCP table was re-run on 2026-11-01 with the existing process-tree harness (`scripts/benchmarks/runtime/mcp-bench.mjs`), not Hyperfine: warm explore 142 ms against 244 ms, the eight-explore wave 1.89 s against 2.40 s, `initialize` 62 ms against 120 ms, and idle memory 2.08 GiB against 4.31 GiB. The watcher sync is 754 ms against 676 ms (12% slower). Keep this table separate from the Hyperfine index/sync figures, as the README does. - **Raw evidence.** `~/cg-scratch/bench-2026-11-01/`: `*-index.json` / `*-sync.json`, `graph.tsv`, `*-index.txt` / `*-sync.txt`, and `admission-*.json`. From 5d8c9901956c7e45509a6d0b82c0509167bc13c9 Mon Sep 17 00:00:00 2001 From: bompus Date: Sun, 1 Nov 2026 11:53:44 -0700 Subject: [PATCH 3/8] docs: make the 2026-11-01 benchmark reproducible Add the Hyperfine whole-command runner, the per-run append-probe helper and the chart generator; the benchmark record now gives the exact commands, the corpus list and how each repeated sync receives a fresh edit (unique marker appended by Hyperfine's --prepare, file restored after each arm). The chart is regenerated from the Hyperfine JSON, and run-index-sync.sh is labelled as the pre-Hyperfine /usr/bin/time harness. Align the README's sync percentages to elapsed-time reductions (18% gin to 62% n8n, with 1.22x-2.63x reciprocal speed) and fix the stale claim that CLI sync wins on all seven corpora; discourse is 3% slower. --- README.md | 6 +- assets/fork-vs-upstream-performance.svg | 38 ++-- .../benchmarks/fork-vs-upstream-2026-11-01.md | 19 ++ scripts/benchmarks/runtime/README.md | 8 +- .../runtime/hyperfine-append-probe.sh | 8 + .../benchmarks/runtime/hyperfine-chart.mjs | 173 ++++++++++++++++++ .../runtime/hyperfine-index-sync.sh | 45 +++++ 7 files changed, 273 insertions(+), 24 deletions(-) create mode 100644 scripts/benchmarks/runtime/hyperfine-append-probe.sh create mode 100644 scripts/benchmarks/runtime/hyperfine-chart.mjs create mode 100644 scripts/benchmarks/runtime/hyperfine-index-sync.sh diff --git a/README.md b/README.md index 9107410cc..d850ed8ce 100644 --- a/README.md +++ b/README.md @@ -200,11 +200,11 @@ Call resolution the fork adds: Upstream `main` at `cd04b21a` against the fork at `8313e9a0`, Node.js 24.22.2 on the same 24-core bare-metal Linux host, measured 2026-11-01 under the host benchmark lock with Hyperfine 2.0.0. Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked no clear change rather than given a fixed band. Bun was not re-measured in this run; the Bun figures below are from the 2026-10-28 campaign, and the CLI table is from the 2026-10-27 run at `bc27fdb4`. The numbers, method, spreads, graph sizes and admission receipts are in [`docs/benchmarks/fork-vs-upstream-2026-11-01.md`](docs/benchmarks/fork-vs-upstream-2026-11-01.md). The scripts are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). -![Fork against upstream CodeGraph on seven corpora, Node, 2026-11-01: the fork syncs six of seven one-file edits 1.22 to 2.63 times faster (discourse 3% slower), indexes CPython 1.79 times faster and n8n 1.43 times faster, and is level with gin and supabase, the latter on 18.5% more nodes.](assets/fork-vs-upstream-performance.svg?v=14) +![Fork against upstream CodeGraph on seven corpora, Node, 2026-11-01: the fork syncs six of seven one-file edits 1.22 to 2.63 times faster (discourse 3% slower), indexes CPython 1.79 times faster and n8n 1.43 times faster, and is level with gin and supabase, the latter on 18.5% more nodes.](assets/fork-vs-upstream-performance.svg?v=15) *2026-11-01 Node run, fork `8313e9a0` against upstream `cd04b21a`. Ratios are exact; the tables mark which gaps clear the run-to-run spread.* -The fork's one-file sync is faster on six of seven corpora: 22% (gin) to 163% (n8n), or 1.22× to 2.63×; discourse is 3% slower. Full indexing: CPython is 1.79× faster (25.84 s against 46.36 s), n8n 1.43× faster (50.16 s against 71.75 s), discourse 8% faster, Alamofire 5% faster and pretix 1% faster; gin and supabase are no clear change. Supabase is 1.9% slower inside a 2.1% fork spread, on a graph with 18.5% more nodes and 6.7% more edges. The speed costs memory on the smaller corpora — index peaks +49% on gin, +34% on Alamofire and +16% on pretix — while it is lower on CPython (−20%), discourse (−8%), supabase (−1%) and n8n (−29%). MCP on n8n: warm explore 1.72× faster, the eight-explore wave 1.27× faster and `initialize` 1.94× faster, with busy memory 19% lower and idle memory 52% lower; the watcher's one-file re-index is the one loss, 12% slower (754 ms against 676 ms). +The fork's one-file sync is faster on six of seven corpora: 18% (gin) to 62% (n8n) less elapsed time, or 1.22× to 2.63× reciprocal speed; discourse is 3% slower. Full indexing: CPython is 1.79× faster (25.84 s against 46.36 s), n8n 1.43× faster (50.16 s against 71.75 s), discourse 8% faster, Alamofire 5% faster and pretix 1% faster; gin and supabase are no clear change. Supabase is 1.9% slower inside a 2.1% fork spread, on a graph with 18.5% more nodes and 6.7% more edges. The speed costs memory on the smaller corpora — index peaks +49% on gin, +34% on Alamofire and +16% on pretix — while it is lower on CPython (−20%), discourse (−8%), supabase (−1%) and n8n (−29%). MCP on n8n: warm explore 1.72× faster, the eight-explore wave 1.27× faster and `initialize` 1.94× faster, with busy memory 19% lower and idle memory 52% lower; the watcher's one-file re-index is the one loss, 12% slower (754 ms against 676 ms). The [2026-10-28 Node+Bun campaign](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md), the [2026-10-30 Node re-measurement](docs/benchmarks/fork-vs-upstream-node-2026-10-30.md), the [first 2026-10-31 run](docs/benchmarks/fork-vs-upstream-2026-10-31.md) and the [second 2026-10-31 run](docs/benchmarks/fork-vs-upstream-2026-10-31b.md) are superseded by this run and kept with their original dates and commits. @@ -284,7 +284,7 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne - **Larger database:** 11–20% bigger on six of the seven corpora above, and 31.5% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. - **Fewer edges on some projects:** 7% fewer on pretix and 1.7% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. - **More memory while indexing on the smaller projects:** in the 2026-11-01 comparison the fork's full index peaks higher on gin (+49%), Alamofire (+34%) and pretix (+16%) — the in-process resolver-pool workers used on small and mid-size projects — while it is lower on CPython (−20%), discourse (−8%), supabase (−1%) and n8n (−29%). -- **MCP watcher sync:** the daemon's one-file re-index is 9% slower (766 ms against 702 ms) because it also runs the deferred synthesized-edge refresh; the CLI one-file sync is faster on all seven corpora. +- **MCP watcher sync:** the daemon's one-file re-index is 9% slower (766 ms against 702 ms) because it also runs the deferred synthesized-edge refresh; the CLI one-file sync is faster on six of the seven corpora (discourse is 3% slower). --- diff --git a/assets/fork-vs-upstream-performance.svg b/assets/fork-vs-upstream-performance.svg index c301cbfcc..6b8ae9daf 100644 --- a/assets/fork-vs-upstream-performance.svg +++ b/assets/fork-vs-upstream-performance.svg @@ -7,10 +7,10 @@ Upstream cd04b21a · fork 8313e9a0 · Node 24.22.2 · Hyperfine 2.0.0 Index: median of 2 runs, fork / upstream (lower is better). Sync: median of 4 runs, upstream / fork (higher is better). Full index - -0.5× - -0.75× + +0.5× + +0.75× 1× @@ -18,8 +18,8 @@ 1.5× gin (Go) - -0.99× + +0.99× Alamofire (Swift) 0.95× @@ -39,8 +39,8 @@ 0.70× One-file sync - -0.5× + +0.5× 1× @@ -49,26 +49,26 @@ 2× 2.5× - -3× + +3× gin (Go) - -1.22× + +1.22× Alamofire (Swift) - -1.40× + +1.40× pretix (Python) - -1.68× + +1.68× CPython (C/Py) -2.02× +2.01× discourse (Ruby) 0.97× supabase (TS) - -2.20× + +2.20× n8n (Vue/TS) 2.63× diff --git a/docs/benchmarks/fork-vs-upstream-2026-11-01.md b/docs/benchmarks/fork-vs-upstream-2026-11-01.md index 7f050c4f6..6caad17fa 100644 --- a/docs/benchmarks/fork-vs-upstream-2026-11-01.md +++ b/docs/benchmarks/fork-vs-upstream-2026-11-01.md @@ -41,6 +41,25 @@ The supabase node count fell from 152,274 to 135,658 across #612 and #613; the i The fork syncs faster on six of seven corpora. Discourse is 3% slower, the one sync loss; its 2.9% gap clears the larger arm spread (1.8%). The other six clear their spreads too. +## Reproduce + +The campaign runner is [`scripts/benchmarks/runtime/hyperfine-index-sync.sh`](../../../scripts/benchmarks/runtime/hyperfine-index-sync.sh) with [`hyperfine-append-probe.sh`](../../../scripts/benchmarks/runtime/hyperfine-append-probe.sh); the chart and summary are generated by [`hyperfine-chart.mjs`](../../../scripts/benchmarks/runtime/hyperfine-chart.mjs). This is separate from the earlier `run-index-sync.sh`, which uses `/usr/bin/time` and is not comparable. + +```bash +export BENCH_OUT=$HOME/cg-scratch/bench-2026-11-01 +export BENCH_ARMS="upstream=$(fnm exec --using codegraph which node)@/path/to/upstream-checkout fork=$(fnm exec --using codegraph which node)@/path/to/fork-checkout" +bash scripts/benchmarks/runtime/hyperfine-index-sync.sh index-sync.list +node scripts/benchmarks/runtime/hyperfine-chart.mjs "$BENCH_OUT" \ + --svg assets/fork-vs-upstream-performance.svg \ + --date 2026-11-01 --upstream cd04b21a --fork 8313e9a0 +``` + +`index-sync.list` is one corpus per line, ` `: the seven entries in `~/cg-scratch/index-sync-2026-10-28.list` (gin `auth.go` `//`, Alamofire `Source/Alamofire.swift` `//`, pretix `src/pretix/base/addressvalidation.py` `#`, CPython `Lib/__future__.py` `#`, discourse `config/boot.rb` `#`, supabase `apps/studio/lib/base64url.ts` `//`, n8n `packages/cli/src/abstract-server.ts` `//`). + +Each invocation runs one Hyperfine index command with `--prepare 'rm -rf .cg-bench'`, then two sync commands against that index. Every sync run's `--prepare` calls `hyperfine-append-probe.sh`, which appends a unique comment plus `void_probe_marker_ = 1`; the runner restores the file with `git checkout` after each arm. That is how repeated sync runs each see a fresh one-file edit. Peak RSS is Hyperfine's `memory_peak_resident`. + +The campaign wrapped each corpus in the host benchmark lock and a fresh 10-second admission (`flock ~/.cache/setupLinuxHost/post-upgrade-bench.lock`, then `python3 admission.py`); the runner itself takes no lock. + ## Notes - **Runner change.** This is the first campaign timed with Hyperfine 2.0.0 end to end. Every arm was re-run; the 2026-10-31 figures are not directly comparable and should not be mixed with these. diff --git a/scripts/benchmarks/runtime/README.md b/scripts/benchmarks/runtime/README.md index 4582f26e9..e50f836cf 100644 --- a/scripts/benchmarks/runtime/README.md +++ b/scripts/benchmarks/runtime/README.md @@ -7,7 +7,9 @@ The arms can differ in build, in runtime, or in both. | Script | Measures | Output in `$BENCH_OUT` | |---|---|---| -| `run-index-sync.sh ` | Full index, then two one-file syncs: wall time, peak RSS, CPU, database size, node and edge counts | `index-sync.tsv`, `logs/` | +| `run-index-sync.sh ` | Pre-Hyperfine harness: full index, then two one-file syncs with `/usr/bin/time`: wall time, peak RSS, CPU, database size, node and edge counts. Not comparable with the Hyperfine rows. | `index-sync.tsv`, `logs/` | +| `hyperfine-index-sync.sh ` | Full index and two one-file syncs timed with Hyperfine whole commands, mirrored arms; peak RSS from Hyperfine's JSON, graph counts from the fresh index | `*-index.json`, `*-sync.json`, `graph.tsv` | +| `hyperfine-chart.mjs [--svg … --date … --upstream … --fork …]` | The README chart and a Markdown summary (medians, spreads, verdicts) from `hyperfine-index-sync.sh` output | SVG + stdout | | `run-mcp.sh ` | MCP server: start to first explore answer, warm explore median and p90, 8 explores at once, watcher sync, idle memory and CPU | `mcp.jsonl` | | `run-cli.sh ` | CLI start: `--version`, `status`, one `explore` (hyperfine, 20 runs) | `cli-*.json`, `cli.txt` | | `sync-ab.mjs --corpus … --arm base=… --arm cand=…` | One change on one corpus: a one-file sync before and after, with same-build controls, until the result qualifies or stays inside run-to-run variation (see below) | `runs.jsonl`, `summary.md`, `summary.json`, `chart.json`, `logs/` in `--out` | @@ -42,7 +44,9 @@ export BENCH_WRAP="nice -n 10 systemd-run --user --scope -p MemoryMax=16G --" `BENCH_ARMS` entries are `name=RUNTIME@BUILD`. `BENCH_WRAP` prefixes every timed command; it defaults to `nice -n 10`. `run-mcp.sh` runs `mcp-bench.mjs` -with `$BENCH_NODE` (default `node`, 22.13 or newer). +with `$BENCH_NODE` (default `node`, 22.13 or newer). `hyperfine-index-sync.sh` +takes the same arms and list; it does not wait for a quiet host or take the +benchmark lock, so wrap it in the host's admission and lock for a campaign. ## Corpus lists diff --git a/scripts/benchmarks/runtime/hyperfine-append-probe.sh b/scripts/benchmarks/runtime/hyperfine-append-probe.sh new file mode 100644 index 000000000..821d8e7f4 --- /dev/null +++ b/scripts/benchmarks/runtime/hyperfine-append-probe.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# Append a unique one-file change for one Hyperfine sync run. The timestamp is +# the new content, so every repetition of `codegraph sync` sees a real edit; +# the caller restores the file after the arm. +set -u +file=$1; cm=$2 +n=$(date +%s%N) +printf '\n%s cg bench probe %s\nvoid_probe_marker_%s = 1\n' "$cm" "$n" "$n" >> "$file" diff --git a/scripts/benchmarks/runtime/hyperfine-chart.mjs b/scripts/benchmarks/runtime/hyperfine-chart.mjs new file mode 100644 index 000000000..a38b131c9 --- /dev/null +++ b/scripts/benchmarks/runtime/hyperfine-chart.mjs @@ -0,0 +1,173 @@ +#!/usr/bin/env node +// Render the README chart and a Markdown summary from a Hyperfine campaign +// directory produced by hyperfine-index-sync.sh. +// +// usage: node hyperfine-chart.mjs [--svg ] [--date YYYY-MM-DD] +// [--upstream ] [--fork ] +// +// Files are named ---.json by the runner. Peak RSS +// is Hyperfine's memory_peak_resident. A comparison counts as a change only +// when the fork/upstream gap exceeds the larger arm's (max-min)/median spread, +// matching the benchmark record's rule. + +import { readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { basename, join } from 'node:path'; + +const args = process.argv.slice(2); +const dir = args.shift(); +if (!dir) { + console.error('usage: hyperfine-chart.mjs [--svg ] [--date YYYY-MM-DD] [--upstream ] [--fork ]'); + process.exit(2); +} +const flag = (name) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : undefined; +}; +const date = flag('--date') ?? new Date().toISOString().slice(0, 10); +const upstream = flag('--upstream') ?? 'upstream'; +const fork = flag('--fork') ?? 'fork'; +const svgPath = flag('--svg') ?? 'assets/fork-vs-upstream-performance.svg'; + +const LABELS = { + gin: 'gin (Go)', + alamofire: 'Alamofire (Swift)', + pretix: 'pretix (Python)', + cpython: 'CPython (C/Py)', + discourse: 'discourse (Ruby)', + supabase: 'supabase (TS)', + n8n: 'n8n (Vue/TS)', +}; +const ORDER = ['gin', 'alamofire', 'pretix', 'cpython', 'discourse', 'supabase', 'n8n']; +const FILE = /^(.*)-(upstream|fork)-\d+-(index|sync)\.json$/; + +const samples = new Map(); // corpus -> arm -> kind -> { times, rss } +for (const file of readdirSync(dir)) { + const m = FILE.exec(file); + if (!m) continue; + const [, corpus, arm, kind] = m; + const data = JSON.parse(readFileSync(join(dir, file), 'utf8')); + for (const result of data.results ?? []) { + for (const run of result.measurements ?? []) { + const key = `${corpus}\0${arm}\0${kind}`; + const slot = samples.get(key) ?? { times: [], rss: [] }; + slot.times.push(run.time_wall_clock.value); + slot.rss.push(run.memory_peak_resident.value / 2 ** 20); + samples.set(key, slot); + } + } +} + +function stats(corpus, arm, kind) { + const s = samples.get(`${corpus}\0${arm}\0${kind}`); + if (!s || s.times.length === 0) return null; + const sorted = [...s.times].sort((a, b) => a - b); + const mid = sorted.length >> 1; + const median = sorted.length % 2 === 1 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; + return { + median, + min: sorted[0], + max: sorted[sorted.length - 1], + spread: sorted[sorted.length - 1] > 0 ? (sorted[sorted.length - 1] - sorted[0]) / median : 0, + rss: Math.max(...s.rss), + n: sorted.length, + }; +} + +const corpora = ORDER.filter((c) => stats(c, 'upstream', 'index') && stats(c, 'fork', 'index')); +if (corpora.length === 0) { + console.error(`no ${basename(dir)} Hyperfine results found`); + process.exit(1); +} + +const fmt = (n, digits = 2) => n.toFixed(digits); +const pct = (n) => `${(Math.abs(n) * 100).toFixed(0)}%`; +function verdict(ratio, a, b, higherIsBetter = false) { + const spread = Math.max(a.spread, b.spread); + const gap = Math.abs(ratio - 1); + if (gap <= spread) return 'no clear change'; + // Report elapsed-time change: for sync the ratio is upstream/fork, so a + // ratio of 1.22 is 18% less time, not a 22% time reduction. + const faster = higherIsBetter ? ratio > 1 : ratio < 1; + const elapsed = higherIsBetter + ? (ratio > 1 ? 1 - 1 / ratio : 1 / ratio - 1) + : Math.abs(1 - ratio); + return `${pct(elapsed)} ${faster ? 'faster' : 'slower'}`; +} + +console.log('| Corpus | Index upstream | Index fork | Fork / upstream | Verdict | Sync upstream | Sync fork | Upstream / fork | Verdict |'); +console.log('|---|---:|---:|---:|---|---:|---:|---:|---|'); +for (const corpus of corpora) { + const ui = stats(corpus, 'upstream', 'index'); + const fi = stats(corpus, 'fork', 'index'); + const us = stats(corpus, 'upstream', 'sync'); + const fs = stats(corpus, 'fork', 'sync'); + const ir = fi.median / ui.median; + const sr = fs ? us.median / fs.median : NaN; + console.log( + `| ${corpus} | ${fmt(ui.median, 3)} s | ${fmt(fi.median, 3)} s | ${fmt(ir, 3)} | ${verdict(ir, ui, fi)} | ` + + `${fmt(us.median, 3)} s | ${fmt(fs.median, 3)} s | ${fmt(sr, 3)} | ${verdict(sr, us, fs, true)} |`, + ); +} + +// Chart geometry copied from the 2026-10-31 chart: index panel 0.5x-1.5x, +// sync panel 0.5x-3x, seven rows. +const indexX1 = 241.9; +const indexScale = 391.8; +const syncX1 = 609.2; +const syncScale = 172.4; +function bar(x1, scale, ratio) { + if (ratio <= 1) { + const w = (1 - ratio) * scale; + return { x: x1 - w, w, tx: x1 - w - 6, anchor: 'end' }; + } + const w = (ratio - 1) * scale; + return { x: x1, w, tx: x1 + w + 6, anchor: 'start' }; +} +const lines = []; +lines.push(''); +lines.push('Fork versus upstream on seven corpora, Node, ' + date + ''); +lines.push('Two panels: full index fork/upstream ratios and one-file sync upstream/fork speedups for gin, Alamofire, pretix, CPython, discourse, supabase and n8n.'); +lines.push(''); +lines.push(''); +lines.push('Fork vs upstream on Node — seven corpora, ' + date + ''); +lines.push('Upstream ' + upstream + ' · fork ' + fork + ' · Node 24.22.2 · Hyperfine 2.0.0'); +lines.push('Index: median of 2 runs, fork / upstream (lower is better). Sync: median of 4 runs, upstream / fork (higher is better).'); +lines.push('Full index'); +for (const [label, x, muted] of [['0.5×', 46.0, false], ['0.75×', 144.0, false], ['1×', indexX1, true], ['1.25×', 339.9, false], ['1.5×', 437.8, false]]) { + lines.push(``); + lines.push(`${label}`); +} +let y = 200; +for (const corpus of corpora) { + const ui = stats(corpus, 'upstream', 'index'); + const fi = stats(corpus, 'fork', 'index'); + const ratio = fi.median / ui.median; + const b = bar(indexX1, indexScale, ratio); + lines.push(`${LABELS[corpus] ?? corpus}`); + lines.push(``); + lines.push(`${fmt(ratio)}×`); + y += 58; +} +lines.push('One-file sync'); +for (const [label, x, muted] of [['0.5×', 523.0, false], ['1×', syncX1, true], ['1.5×', 695.4, false], ['2×', 781.6, false], ['2.5×', 867.8, false], ['3×', 954.0, false]]) { + lines.push(``); + lines.push(`${label}`); +} +y = 200; +for (const corpus of corpora) { + const us = stats(corpus, 'upstream', 'sync'); + const fs = stats(corpus, 'fork', 'sync'); + const ratio = us.median / fs.median; + const b = bar(syncX1, syncScale, ratio); + const cls = ratio >= 1 ? 'good' : 'bad'; + lines.push(`${LABELS[corpus] ?? corpus}`); + lines.push(``); + lines.push(`${fmt(ratio)}×`); + y += 58; +} +lines.push('Exact ratios; the tables mark which gaps clear the larger arm\'s run-to-run spread.'); +lines.push(`Method, spreads, graph sizes, admission receipts: docs/benchmarks/fork-vs-upstream-${date}.md`); +lines.push('CodeGraph fork · bompus/codegraph · measured under the host benchmark lock'); +lines.push(''); +writeFileSync(svgPath, lines.join('\n') + '\n'); +console.log(`wrote ${svgPath}`); diff --git a/scripts/benchmarks/runtime/hyperfine-index-sync.sh b/scripts/benchmarks/runtime/hyperfine-index-sync.sh new file mode 100644 index 000000000..e4aca2775 --- /dev/null +++ b/scripts/benchmarks/runtime/hyperfine-index-sync.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# Hyperfine whole-command runner for the seven-corpus index/sync campaign. +# +# usage: BENCH_ARMS="upstream=@ fork=@" \ +# BENCH_OUT= \ +# hyperfine-index-sync.sh +# +# Each list line: . +# For each corpus the arms run in mirrored order (upstream, fork, fork, +# upstream). Each invocation builds a fresh .cg-bench with one Hyperfine run +# and records the graph counts, then times two `codegraph sync` runs against +# that index. Hyperfine's --prepare appends a unique marker before every sync +# (hyperfine-append-probe.sh), so each run sees a one-file change; the file is +# restored after the arm. +# +# Peak RSS comes from Hyperfine's memory_peak_resident in the exported JSON. +# This script does no host admission or locking; the campaign wraps it in the +# host's benchmark lock and admission check. +set -u +HERE=$(cd "$(dirname "$0")" && pwd); . "$HERE/arms.sh" +while read -r corpus file cm; do + [ -z "$corpus" ] || [ "${corpus:0:1}" = "#" ] && continue + NAME=$(basename "$corpus") + cd "$corpus" || exit 1 + for arm in "${MIRRORED[@]}"; do + runtime=${RUNTIME[$arm]}; build=${BUILD[$arm]} + rm -rf .cg-bench + hyperfine --warmup 0 --runs 1 --style basic --prepare 'rm -rf .cg-bench' \ + --export-json "$BENCH_OUT/$NAME-$arm-$(date +%s%N)-index.json" \ + "env CODEGRAPH_TELEMETRY=0 DO_NOT_TRACK=1 CODEGRAPH_NO_DAEMON=1 CODEGRAPH_ALLOW_UNSAFE_NODE=1 CODEGRAPH_DIR=.cg-bench $runtime $build/dist/bin/codegraph.js init --yes ." \ + >> "$BENCH_OUT/$NAME-index.txt" 2>&1 || exit 7 + db=.cg-bench/codegraph.db + printf '%s\t%s\t%s\t%s\t%s\n' "$NAME" "$arm" \ + "$(sqlite3 "$db" 'select count(*) from nodes')" \ + "$(sqlite3 "$db" 'select count(*) from edges')" \ + "$(stat -c %s "$db")" >> "$BENCH_OUT/graph.tsv" + hyperfine --warmup 0 --runs 2 --style basic \ + --prepare "bash $HERE/hyperfine-append-probe.sh '$file' '$cm'" \ + --export-json "$BENCH_OUT/$NAME-$arm-$(date +%s%N)-sync.json" \ + "env CODEGRAPH_TELEMETRY=0 DO_NOT_TRACK=1 CODEGRAPH_NO_DAEMON=1 CODEGRAPH_ALLOW_UNSAFE_NODE=1 CODEGRAPH_DIR=.cg-bench $runtime $build/dist/bin/codegraph.js sync ." \ + >> "$BENCH_OUT/$NAME-sync.txt" 2>&1 || exit 7 + git checkout -q -- "$file" + rm -rf .cg-bench + done +done From 9823669e0a0256999fc39b4e6cd39e88639ddd44 Mon Sep 17 00:00:00 2001 From: bompus Date: Sun, 1 Nov 2026 12:15:48 -0700 Subject: [PATCH 4/8] fix(bench): harden the Hyperfine campaign runner and chart Runner: read the list file it advertises; reject a dirty probe and keep its original bytes in a temp file restored after every arm and in an EXIT/INT/TERM trap; reject a non-fresh BENCH_OUT; apply the BENCH_WRAP prefix to every timed command; require the probe in the fresh index and verify the last sync stored the last appended bytes by comparing the file SHA-256 with files.content_hash. Chart: accept --arms base,candidate; match corpus names case-insensitively and require the expected corpora; reject a sample count that does not match one campaign (2 index / 4 sync by default); derive the qualification gap from the fork/base elapsed ratio for both metrics. Record: list each corpus commit and edited file, and describe the new guards. The chart is regenerated from the Hyperfine JSON and the README image tag is bumped to v16. --- README.md | 2 +- assets/fork-vs-upstream-performance.svg | 6 +- .../benchmarks/fork-vs-upstream-2026-11-01.md | 14 ++- scripts/benchmarks/runtime/README.md | 2 +- .../benchmarks/runtime/hyperfine-chart.mjs | 119 ++++++++++++------ .../runtime/hyperfine-index-sync.sh | 75 ++++++++--- 6 files changed, 161 insertions(+), 57 deletions(-) diff --git a/README.md b/README.md index d850ed8ce..9c1cdac22 100644 --- a/README.md +++ b/README.md @@ -200,7 +200,7 @@ Call resolution the fork adds: Upstream `main` at `cd04b21a` against the fork at `8313e9a0`, Node.js 24.22.2 on the same 24-core bare-metal Linux host, measured 2026-11-01 under the host benchmark lock with Hyperfine 2.0.0. Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked no clear change rather than given a fixed band. Bun was not re-measured in this run; the Bun figures below are from the 2026-10-28 campaign, and the CLI table is from the 2026-10-27 run at `bc27fdb4`. The numbers, method, spreads, graph sizes and admission receipts are in [`docs/benchmarks/fork-vs-upstream-2026-11-01.md`](docs/benchmarks/fork-vs-upstream-2026-11-01.md). The scripts are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). -![Fork against upstream CodeGraph on seven corpora, Node, 2026-11-01: the fork syncs six of seven one-file edits 1.22 to 2.63 times faster (discourse 3% slower), indexes CPython 1.79 times faster and n8n 1.43 times faster, and is level with gin and supabase, the latter on 18.5% more nodes.](assets/fork-vs-upstream-performance.svg?v=15) +![Fork against upstream CodeGraph on seven corpora, Node, 2026-11-01: the fork syncs six of seven one-file edits 1.22 to 2.63 times faster (discourse 3% slower), indexes CPython 1.79 times faster and n8n 1.43 times faster, and is level with gin and supabase, the latter on 18.5% more nodes.](assets/fork-vs-upstream-performance.svg?v=16) *2026-11-01 Node run, fork `8313e9a0` against upstream `cd04b21a`. Ratios are exact; the tables mark which gaps clear the run-to-run spread.* diff --git a/assets/fork-vs-upstream-performance.svg b/assets/fork-vs-upstream-performance.svg index 6b8ae9daf..40b6e74b7 100644 --- a/assets/fork-vs-upstream-performance.svg +++ b/assets/fork-vs-upstream-performance.svg @@ -1,11 +1,11 @@ Fork versus upstream on seven corpora, Node, 2026-11-01 -Two panels: full index fork/upstream ratios and one-file sync upstream/fork speedups for gin, Alamofire, pretix, CPython, discourse, supabase and n8n. +Two panels: full index candidate/base ratios and one-file sync base/candidate speedups for gin, Alamofire, pretix, CPython, discourse, supabase and n8n. Fork vs upstream on Node — seven corpora, 2026-11-01 -Upstream cd04b21a · fork 8313e9a0 · Node 24.22.2 · Hyperfine 2.0.0 -Index: median of 2 runs, fork / upstream (lower is better). Sync: median of 4 runs, upstream / fork (higher is better). +Upstream cd04b21a · Fork 8313e9a0 · Node 24.22.2 · Hyperfine 2.0.0 +Index: median of 2 runs, candidate / base (lower is better). Sync: median of 4 runs, base / candidate (higher is better). Full index 0.5× diff --git a/docs/benchmarks/fork-vs-upstream-2026-11-01.md b/docs/benchmarks/fork-vs-upstream-2026-11-01.md index 6caad17fa..109240caf 100644 --- a/docs/benchmarks/fork-vs-upstream-2026-11-01.md +++ b/docs/benchmarks/fork-vs-upstream-2026-11-01.md @@ -56,7 +56,19 @@ node scripts/benchmarks/runtime/hyperfine-chart.mjs "$BENCH_OUT" \ `index-sync.list` is one corpus per line, ` `: the seven entries in `~/cg-scratch/index-sync-2026-10-28.list` (gin `auth.go` `//`, Alamofire `Source/Alamofire.swift` `//`, pretix `src/pretix/base/addressvalidation.py` `#`, CPython `Lib/__future__.py` `#`, discourse `config/boot.rb` `#`, supabase `apps/studio/lib/base64url.ts` `//`, n8n `packages/cli/src/abstract-server.ts` `//`). -Each invocation runs one Hyperfine index command with `--prepare 'rm -rf .cg-bench'`, then two sync commands against that index. Every sync run's `--prepare` calls `hyperfine-append-probe.sh`, which appends a unique comment plus `void_probe_marker_ = 1`; the runner restores the file with `git checkout` after each arm. That is how repeated sync runs each see a fresh one-file edit. Peak RSS is Hyperfine's `memory_peak_resident`. +The corpus checkouts were clean at these commits; the edited file was restored after every arm. Untracked benchmark index directories (`.cg-*`, `.codegraph`) are omitted and do not affect the source tree. + +| Corpus | Commit | Edited file | +|---|---|---| +| gin | `1bd0ecf` | `auth.go` | +| Alamofire | `bda9ed5` | `Source/Alamofire.swift` | +| pretix | `aa14505` | `src/pretix/base/addressvalidation.py` | +| CPython | `1e8ff18a` | `Lib/__future__.py` | +| discourse | `ae5ba2c4` | `config/boot.rb` | +| supabase | `36371de1` | `apps/studio/lib/base64url.ts` | +| n8n | `7cc0450a` | `packages/cli/src/abstract-server.ts` | + +Each invocation runs one Hyperfine index command with `--prepare 'rm -rf .cg-bench'`, then two sync commands against that index. Every sync run's `--prepare` calls `hyperfine-append-probe.sh`, which appends a unique comment plus `void_probe_marker_ = 1`; the runner restores the file with `git checkout` after each arm. That is how repeated sync runs each see a fresh one-file edit. The runner rejects a probe with uncommitted changes, saves its original bytes outside the repository and restores them on success, failure and interruption. It also rejects a `BENCH_OUT` that already holds a Hyperfine campaign, verifies the probe is in each fresh index, and checks that the last sync stored the last appended bytes; the chart rejects a sample count that does not match one campaign. Peak RSS is Hyperfine's `memory_peak_resident`. The campaign wrapped each corpus in the host benchmark lock and a fresh 10-second admission (`flock ~/.cache/setupLinuxHost/post-upgrade-bench.lock`, then `python3 admission.py`); the runner itself takes no lock. diff --git a/scripts/benchmarks/runtime/README.md b/scripts/benchmarks/runtime/README.md index e50f836cf..04b290d1b 100644 --- a/scripts/benchmarks/runtime/README.md +++ b/scripts/benchmarks/runtime/README.md @@ -9,7 +9,7 @@ The arms can differ in build, in runtime, or in both. |---|---|---| | `run-index-sync.sh ` | Pre-Hyperfine harness: full index, then two one-file syncs with `/usr/bin/time`: wall time, peak RSS, CPU, database size, node and edge counts. Not comparable with the Hyperfine rows. | `index-sync.tsv`, `logs/` | | `hyperfine-index-sync.sh ` | Full index and two one-file syncs timed with Hyperfine whole commands, mirrored arms; peak RSS from Hyperfine's JSON, graph counts from the fresh index | `*-index.json`, `*-sync.json`, `graph.tsv` | -| `hyperfine-chart.mjs [--svg … --date … --upstream … --fork …]` | The README chart and a Markdown summary (medians, spreads, verdicts) from `hyperfine-index-sync.sh` output | SVG + stdout | +| `hyperfine-chart.mjs [--svg … --date … --upstream … --fork … --arms base,candidate]` | The README chart and a Markdown summary (medians, spreads, verdicts) from `hyperfine-index-sync.sh` output. Arm names default to `upstream,fork`; corpus names are matched case-insensitively, and a sample count that does not match one campaign is rejected | SVG + stdout | | `run-mcp.sh ` | MCP server: start to first explore answer, warm explore median and p90, 8 explores at once, watcher sync, idle memory and CPU | `mcp.jsonl` | | `run-cli.sh ` | CLI start: `--version`, `status`, one `explore` (hyperfine, 20 runs) | `cli-*.json`, `cli.txt` | | `sync-ab.mjs --corpus … --arm base=… --arm cand=…` | One change on one corpus: a one-file sync before and after, with same-build controls, until the result qualifies or stays inside run-to-run variation (see below) | `runs.jsonl`, `summary.md`, `summary.json`, `chart.json`, `logs/` in `--out` | diff --git a/scripts/benchmarks/runtime/hyperfine-chart.mjs b/scripts/benchmarks/runtime/hyperfine-chart.mjs index a38b131c9..32e150f12 100644 --- a/scripts/benchmarks/runtime/hyperfine-chart.mjs +++ b/scripts/benchmarks/runtime/hyperfine-chart.mjs @@ -3,19 +3,25 @@ // directory produced by hyperfine-index-sync.sh. // // usage: node hyperfine-chart.mjs [--svg ] [--date YYYY-MM-DD] -// [--upstream ] [--fork ] +// [--upstream ] [--fork ] [--arms ,] +// [--base-label