From 2b4758a665d8c6e93f6a849502dbbb97f8a4928a Mon Sep 17 00:00:00 2001 From: bompus Date: Wed, 28 Oct 2026 07:12:40 -0600 Subject: [PATCH 1/2] docs: 2026-10-28 re-measure at de5a7b80 across all seven corpora Index and sync tables regenerated from the full 168-row re-run (upstream 69af19ac, 4 arms x 2 mirrored passes). Fork now wins 6/7 Node index cells outright and 6/7 sync cells per runtime; pretix-Bun index resolves to a tie. New disclosed losses: gin index (~0.1s fixed overhead on a 119-file repo) and discourse sync (~0.5s). Follow-up table gains #573-#575 rows. --- README.md | 36 ++++++------- .../fork-vs-upstream-node-bun-2026-10-27.md | 51 ++++++++++--------- 2 files changed, 45 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 3cd68050e..07e5a2c34 100644 --- a/README.md +++ b/README.md @@ -198,11 +198,11 @@ Call resolution the fork adds: ### Measured results -Upstream `main` at `69af19ac` against the fork at `645abbb8`, each run on Node.js 24.22.1 and on Bun 1.4.4. Index and sync figures were re-measured 2026-10-27 on the same 16-vCPU WSL2 host after the ambient-`declare module` scan fix and the resolve-pipeline changes below; MCP, command-line and explore figures are carried from the earlier 2026-10-27 run at `bc27fdb4` (unchanged by resolution-path changes). Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). +Upstream `main` at `69af19ac` against the fork at `de5a7b80`, each run on Node.js 24.22.1 and on Bun 1.4.4. Index and sync figures were re-measured 2026-10-28 on the same 16-vCPU WSL2 host after the ambient-`declare module` scan fix and the resolve-pipeline changes below; MCP, command-line and explore figures are carried from the earlier 2026-10-27 run at `bc27fdb4` (unchanged by resolution-path changes). Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). ![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 1.5 times faster, syncs an edited file 1.3 times faster, answers a warm question 1.8 times faster and uses 33% less memory while indexing. The fork is better in 57 of 76 comparisons, shows no clear change in 10 and is worse in 9.](assets/fork-vs-upstream-performance.svg?v=8) -The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 4% (pretix) to 56% (CPython), with supabase the one no-clear-change — and on Bun it now wins every clear comparison, pretix included (0.68 s against ~1.13 s after the retry fix below). Full indexing: the fork wins on n8n (1.5×), CPython (2.2×), discourse (1.4×), pretix-Node (8.14 s against 9.04 s) and the Bun arm of gin, Alamofire, discourse, supabase and n8n; gin-Node, Alamofire-Node and supabase-Node tie inside spread; pretix-Bun is within ~2% of upstream (the last residual). The speed costs some memory: indexing peaks higher on pretix (roughly +22% Node / +26% Bun after the worker-cache halving below) and supabase-Bun — the 12 in-process worker heaps are the disclosed trade for the index-time wins — while of the other five corpora on Node, four index at 4–32% less and Alamofire runs 1% higher (inside its spread). Cells where the fork still trails: pretix-Bun index (~+2%), the pretix and supabase-Bun index-memory cells, and a ~0% gin file-watcher cell inside noise. Three earlier losses — pretix-Bun sync, n8n busy-MCP memory on Node and the gin concurrent-explore wave — are wins in this tree; the post-merge measurements are in the benchmark doc's [follow-up section](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#follow-up-merges). The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. +The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 45% (pretix) to 72% (n8n), with discourse the one loss (+47%) — and on Bun it wins six of seven the same way, discourse again the loss (+31%). Full indexing: the fork wins on n8n (1.7×), CPython (2.1×), discourse (1.6×), pretix-Node (7.48 s against 9.69 s), supabase-Node (15.6 s against 19.6 s) and Alamofire-Node (1.19 s against 1.42 s), and on the Bun arm of Alamofire, pretix (7.15 s against 7.20 s — a tie inside spread), CPython, discourse, supabase and n8n; gin index is the remaining loss (+6% Node / +18% Bun — ~0.05–0.14 s of fixed fork overhead on a 119-file index). The speed costs some memory: indexing peaks higher on pretix (+36% Node / +33% Bun) and supabase (+14% Node / +13% Bun) — the 12 in-process worker heaps are the disclosed trade for the index-time wins — while the fork now peaks lower than upstream on n8n (−30%), CPython (−9%), gin (−33%) and discourse-Bun (−28%). Cells where the fork still trails: gin index on both runtimes, discourse sync on both runtimes, and the pretix and supabase index-memory cells. The post-merge measurements are in the benchmark doc's [follow-up section](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#follow-up-merges). The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. The fork starts its resolver pool at or above 150,000 pending references. Below that, it can start the pool part-way through if at least 20,000 unresolved references remain and the estimated main-thread work is at least 0.4 s. An explicit `CODEGRAPH_PARALLEL_RESOLVE_MIN` replaces both gates. Resolution proceeds in pages of 40,000 references (`CODEGRAPH_RESOLVE_BATCH` overrides), across up to 12 workers (`CODEGRAPH_RESOLVE_WORKERS` sets the count, `0` disables the pool; `CODEGRAPH_NO_PARALLEL_RESOLVE=1` does the same). Against a pre-change baseline build, five tested corpora indexed 11-48% faster at 56-132% more peak memory with identical graphs ([measurements](docs/benchmarks/adaptive-pool-engage-2026-10-15.md)). @@ -212,29 +212,29 @@ A [2026-10-15 mid-size comparison](docs/benchmarks/midsize-sweep-2026-10-15.md) | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin (Go, 119 files) | 0.96 s, 558 MiB | 1.02 s, 399 MiB | 0.88 s, 308 MiB | -| Alamofire (Swift, 129 files) | 1.35 s, 879 MiB | 1.41 s, 891 MiB | 0.94 s, 881 MiB | -| pretix (Python + JS, 1,472 files) | 9.04 s, 3.29 GiB | 8.14 s, 4.17 GiB | 7.49 s, 3.79 GiB | -| CPython (C + Python, 3,708 files) | 55.0 s, 7.15 GiB | 25.2 s, 5.90 GiB | 25.1 s, 5.39 GiB | -| discourse (Ruby + JS, 20,277 files) | 19.6 s, 4.07 GiB | 13.7 s, 3.92 GiB | 12.1 s, 2.97 GiB | -| supabase (React + Next.js + TS, 10,716 files) | 18.2 s, 4.86 GiB | 15.5 s, 5.45 GiB | 14.6 s, 4.73 GiB | -| n8n (Vue + TS, 24,434 files) | 74.9 s, 12.39 GiB | 49.5 s, 8.26 GiB | 45.2 s, 8.28 GiB | +| gin (Go, 119 files) | 0.96 s, 567 MiB | 1.02 s, 381 MiB | 0.91 s, 310 MiB | +| Alamofire (Swift, 129 files) | 1.42 s, 817 MiB | 1.19 s, 1.05 GiB | 1.01 s, 887 MiB | +| pretix (Python + JS, 1,472 files) | 9.69 s, 2.92 GiB | 7.48 s, 3.97 GiB | 7.15 s, 3.85 GiB | +| CPython (C + Python, 3,708 files) | 53.7 s, 6.21 GiB | 25.5 s, 5.64 GiB | 25.6 s, 5.41 GiB | +| discourse (Ruby + JS, 20,277 files) | 21.7 s, 3.59 GiB | 13.2 s, 3.58 GiB | 12.4 s, 2.98 GiB | +| supabase (React + Next.js + TS, 10,716 files) | 19.6 s, 4.23 GiB | 15.6 s, 4.83 GiB | 14.5 s, 4.63 GiB | +| n8n (Vue + TS, 24,434 files) | 83.0 s, 11.00 GiB | 49.4 s, 7.65 GiB | 46.6 s, 8.43 GiB | -The fork's index was slower than upstream's on five of the seven corpora in the first pass of this run — the supabase and n8n regressions traced to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scanned every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic, and the follow-up above lands the rest: a port of upstream's text tokenizer plus a `specifier → files` index, so each import checks only candidate declaration files instead of every `.d.ts`. The re-measured numbers in this table are the result: supabase went from 41.5 s to 15.5 s on Node and n8n from 178 s to 49.5 s — n8n now beats upstream 1.5× and CPython 2.2×. The remaining index losses from that run — pretix +33% and Alamofire +18% on Node, and the Bun arm of supabase and n8n — traced not to the ambient scan but to the resolve loop's batch boundaries: every 5,000-ref page drained the worker pipeline before the next fanned out. This run stacks three fixes for that — 40,000-ref pages with a 12-worker pool cap (same batch-boundary semantics; pool workers resolve against a fixed snapshot), then bounded work-stealing dispatch with earlier batch fan-out and non-blocking worker recycling, then a memoized near-duplicate pass that stopped re-tokenizing bodies it had just signed — and those losses are gone: pretix-Node now wins (8.14 s against 9.04 s), the Bun arms of supabase and n8n win, and pretix-Bun is within ~2% of upstream (the last residual). Later in the same stack, the deferred-decode path stopped turning off on framework projects: a framework `extract()` hook no longer forces every file onto decoded-object extraction — its output rides the kernel-buffer transport and merges post-decode ([#563](https://github.com/bompus/codegraph/pull/563); pretix went from 0 to 1,416 kernel-path files, n8n-Node index RSS −10%). The benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). The trade is memory: indexing peaks higher on pretix and supabase now, lower on the other five, and its one-file sync is faster on six of seven (below). +The fork's index was slower than upstream's on five of the seven corpora in the first pass of this run — the supabase and n8n regressions traced to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scanned every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic, and the follow-up above lands the rest: a port of upstream's text tokenizer plus a `specifier → files` index, so each import checks only candidate declaration files instead of every `.d.ts`. The re-measured numbers in this table are the result: supabase went from 41.5 s to 15.5 s on Node and n8n from 178 s to 49.5 s — n8n now beats upstream 1.7× and CPython 2.1×. The remaining index losses from that run — pretix +33% and Alamofire +18% on Node, and the Bun arm of supabase and n8n — traced not to the ambient scan but to the resolve loop's batch boundaries: every 5,000-ref page drained the worker pipeline before the next fanned out. This run stacks three fixes for that — 40,000-ref pages with a 12-worker pool cap (same batch-boundary semantics; pool workers resolve against a fixed snapshot), then bounded work-stealing dispatch with earlier batch fan-out and non-blocking worker recycling, then a memoized near-duplicate pass that stopped re-tokenizing bodies it had just signed — and those losses are gone: pretix-Node wins (7.48 s against 9.69 s), the Bun arms of supabase and n8n win, and pretix-Bun is a tie (7.15 s against 7.20 s). Later in the same stack, the deferred-decode path stopped turning off on framework projects: a framework `extract()` hook no longer forces every file onto decoded-object extraction — its output rides the kernel-buffer transport and merges post-decode ([#563](https://github.com/bompus/codegraph/pull/563); pretix went from 0 to 1,416 kernel-path files, n8n-Node index RSS −10%). The benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). The trade is memory: indexing peaks higher on pretix and supabase now, lower on the other five, and its one-file sync is faster on six of seven (below). -Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s; [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) put the cause in upstream's code). At `69af19ac` upstream's CPython index is slower than at `986de95c` (55.0 s against 40.2 s on this host) while the fork's keeps improving, so the fork's CPython lead is 2.2× this run. +Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s; [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) put the cause in upstream's code). At `69af19ac` upstream's CPython index is slower than at `986de95c` (53.7 s against 40.2 s on this host) while the fork's keeps improving, so the fork's CPython lead is 2.1× this run. **One-file sync** (edit one file, `codegraph sync`): | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin | 0.47 s | 0.40 s | 0.29 s | -| Alamofire | 0.48 s | 0.29 s | 0.18 s | -| pretix | 1.62 s | 1.55 s | 0.68 s | -| CPython | 5.35 s | 2.38 s | 1.66 s | -| discourse | 4.78 s | 2.37 s | 1.77 s | -| supabase | 4.58 s | 2.76 s | 1.95 s | -| n8n | 11.6 s | 9.07 s | 8.10 s | +| gin | 0.34 s | 0.27 s | 0.19 s | +| Alamofire | 0.42 s | 0.27 s | 0.17 s | +| pretix | 1.56 s | 0.86 s | 0.60 s | +| CPython | 6.01 s | 2.65 s | 1.72 s | +| discourse | 1.19 s | 1.75 s | 1.20 s | +| supabase | 8.17 s | 3.05 s | 1.94 s | +| n8n | 11.6 s | 3.29 s | 2.45 s | Upstream's supabase sync works in this run — the `FOREIGN KEY constraint failed` seen in every prior run ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565)) was fixed upstream on 2026-10-19. Its two passes differed widely on Node (1.05 s and 8.1 s); the 4.58 s figure is the median of four, and the spread means that comparison counts as no clear change rather than a fork win. diff --git a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md index 1285e24b6..85809726e 100644 --- a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md +++ b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md @@ -1,12 +1,12 @@ # Fork against upstream CodeGraph on Node.js and Bun, 2026-10-27 -Upstream `main` at `69af19ac` against `fork/consolidated` at `0ee0116c`, each on Node.js 24.22.1 and on Bun 1.4.4, over seven corpora, the MCP server and command-line startup. It repeats the [2026-10-18 run](fork-vs-upstream-node-bun-2026-10-18.md) with the same scripts and the same pinned corpora on the current builds. Index and sync figures were re-measured 2026-10-27 on the same host after the ambient-scan rework, the resolve-pipeline change and the work-stealing dispatch change described in [The supabase resolution regression](#the-supabase-resolution-regression); MCP, CLI and explore cells are unchanged by resolution-path changes and carry the earlier 2026-10-27 measurements at `bc27fdb4`. The README's "Measured results" quote this page. +Upstream `main` at `69af19ac` against `fork/consolidated` at `0ee0116c`, each on Node.js 24.22.1 and on Bun 1.4.4, over seven corpora, the MCP server and command-line startup. It repeats the [2026-10-18 run](fork-vs-upstream-node-bun-2026-10-18.md) with the same scripts and the same pinned corpora on the current builds. Index and sync figures were re-measured **2026-10-28** on the same host at `de5a7b80` — upstream `69af19ac` unchanged — stacking the ambient-scan rework, the resolve-pipeline and work-stealing changes, and the #566–#575 resolver/explore follow-ups described in [Follow-up merges](#follow-up-merges); MCP, CLI and explore cells are unchanged by resolution-path changes and carry the earlier 2026-10-27 measurements at `bc27fdb4`. The README's "Measured results" quote this page. ## Method The scripts are in [`scripts/benchmarks/runtime/`](../../scripts/benchmarks/runtime/): `run-index-sync.sh`, `run-mcp.sh` and `run-cli.sh`, with the four arms below. `adapt-flat-output.mjs` stages the flat output for `render-charts.mjs`, which computes every table, the heatmap and the README card from the raw rows. -- **Builds.** Upstream `main` at `69af19ac` — the commit the fork's default branch is reconciled through — and the fork at `0ee0116c` for index and sync, `bc27fdb4` for MCP, CLI and explore cells. The fork contains every upstream commit through `69af19ac` plus its own work. `bc27fdb4` carried a memo on the per-file ambient-scope scan (without it the fork's supabase index never completed — see [The supabase resolution regression](#the-supabase-resolution-regression)); `eed3d34f` added the text-tokenizer port and the resolve-batch/pool-pipeline change described there; `0ee0116c` adds the work-stealing dispatch, early batch fan-out and fire-and-forget worker recycling described in the same section. Each build was installed with `npm ci`, with the native kernel built from source, and built with `npm run build`. +- **Builds.** Upstream `main` at `69af19ac` — the commit the fork's default branch is reconciled through — and the fork at `de5a7b80` for index and sync, `bc27fdb4` for MCP, CLI and explore cells. The fork contains every upstream commit through `69af19ac` plus its own work. `bc27fdb4` carried a memo on the per-file ambient-scope scan (without it the fork's supabase index never completed — see [The supabase resolution regression](#the-supabase-resolution-regression)); `eed3d34f` added the text-tokenizer port and the resolve-batch/pool-pipeline change described there; `0ee0116c` adds the work-stealing dispatch, early batch fan-out and fire-and-forget worker recycling described in the same section. Each build was installed with `npm ci`, with the native kernel built from source, and built with `npm run build`. - **Arms.** | Arm | Command | @@ -39,24 +39,24 @@ Time and peak memory, median of two. | Corpus | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | | --- | --- | --- | --- | --- | -| gin | 0.96 s, 558 MiB | 1.02 s, 399 MiB | 0.88 s, 308 MiB | 0.96 s, 618 MiB | -| Alamofire | 1.35 s, 879 MiB | 1.41 s, 891 MiB | 0.94 s, 881 MiB | 1.35 s, 974 MiB | -| pretix | 9.04 s, 3.29 GiB | 8.14 s, 4.17 GiB | 7.49 s, 3.79 GiB | 7.20 s, 2.89 GiB | -| CPython | 55.0 s, 7.15 GiB | 25.2 s, 5.90 GiB | 25.1 s, 5.39 GiB | 51.0 s, 6.92 GiB | -| discourse | 19.6 s, 4.07 GiB | 13.7 s, 3.92 GiB | 12.1 s, 2.97 GiB | 18.0 s, 4.25 GiB | -| supabase | 18.2 s, 4.86 GiB | 15.5 s, 5.45 GiB | 14.6 s, 4.73 GiB | 16.4 s, 4.02 GiB | -| n8n | 74.9 s, 12.39 GiB | 49.5 s, 8.26 GiB | 45.2 s, 8.28 GiB | 59.4 s, 9.23 GiB | +| gin | 0.96 s, 567 MiB | 1.02 s, 381 MiB | 0.91 s, 310 MiB | 0.77 s, 587 MiB | +| Alamofire | 1.42 s, 817 MiB | 1.19 s, 1.05 GiB | 1.01 s, 887 MiB | 1.38 s, 976 MiB | +| pretix | 9.69 s, 2.92 GiB | 7.48 s, 3.97 GiB | 7.15 s, 3.85 GiB | 7.20 s, 2.88 GiB | +| CPython | 53.7 s, 6.21 GiB | 25.5 s, 5.64 GiB | 25.6 s, 5.41 GiB | 52.8 s, 7.50 GiB | +| discourse | 21.7 s, 3.59 GiB | 13.2 s, 3.58 GiB | 12.4 s, 2.98 GiB | 19.8 s, 4.15 GiB | +| supabase | 19.6 s, 4.23 GiB | 15.6 s, 4.83 GiB | 14.5 s, 4.63 GiB | 16.8 s, 4.10 GiB | +| n8n | 83.0 s, 11.00 GiB | 49.4 s, 7.65 GiB | 46.6 s, 8.43 GiB | 60.5 s, 9.54 GiB | Graph size. Node and edge counts match across runtimes for each build; database sizes are from the Node runs: | Corpus | Upstream nodes / edges / database | Fork nodes / edges / database | | --- | --- | --- | -| gin | 2,660 / 7,908 / 8.5 MiB | 3,089 / 8,404 / 10.1 MiB | -| Alamofire | 4,600 / 12,516 / 16.2 MiB | 5,614 / 13,682 / 18.3 MiB | -| pretix | 32,781 / 79,684 / 115 MiB | 32,797 / 73,402 / 138 MiB | -| CPython | 167,332 / 615,244 / 492 MiB | 167,923 / 603,861 / 573 MiB | +| gin | 2,660 / 7,908 / 9 MiB | 3,089 / 8,404 / 10 MiB | +| Alamofire | 4,600 / 12,516 / 16 MiB | 5,614 / 13,682 / 18 MiB | +| pretix | 32,783 / 79,686 / 115 MiB | 32,799 / 73,404 / 138 MiB | +| CPython | 167,332 / 615,244 / 492 MiB | 167,923 / 603,830 / 573 MiB | | discourse | 163,411 / 309,678 / 389 MiB | 168,110 / 308,849 / 432 MiB | -| supabase | 114,440 / 292,126 / 335 MiB | 152,274 / 327,838 / 460 MiB | +| supabase | 114,440 / 292,126 / 335 MiB | 152,274 / 327,837 / 460 MiB | | n8n | 315,422 / 1,239,920 / 1.43 GiB | 328,519 / 1,233,163 / 1.67 GiB | The edge counts differ because the builds resolve differently: the fork adds Markdown, route and dispatch links, and declines call links it cannot confirm. The timings compare the cost of building each build's own graph, not the same graph. @@ -67,16 +67,16 @@ Median of four (two syncs per run, two runs). | Corpus | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | | --- | --- | --- | --- | --- | -| gin | 0.47 s | 0.40 s | 0.29 s | 0.29 s | -| Alamofire | 0.48 s | 0.29 s | 0.18 s | 0.31 s | -| pretix | 1.62 s | 1.55 s | 1.27 s | 1.13 s | -| CPython | 5.35 s | 2.38 s | 1.66 s | 3.93 s | -| discourse | 4.78 s | 2.37 s | 1.77 s | 3.23 s | -| supabase | 4.58 s | 2.76 s | 1.95 s | 6.69 s | -| n8n | 11.6 s | 9.07 s | 8.10 s | 8.68 s | +| gin | 0.34 s | 0.27 s | 0.19 s | 0.21 s | +| Alamofire | 0.42 s | 0.27 s | 0.17 s | 0.29 s | +| pretix | 1.56 s | 0.86 s | 0.60 s | 1.11 s | +| CPython | 6.01 s | 2.65 s | 1.72 s | 4.14 s | +| discourse | 1.19 s | 1.75 s | 1.20 s | 0.92 s | +| supabase | 8.17 s | 3.05 s | 1.94 s | 6.70 s | +| n8n | 11.6 s | 3.29 s | 2.45 s | 9.17 s | -Upstream's supabase sync, which failed with `FOREIGN KEY constraint failed` in every prior run, now completes ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565) was fixed upstream on 2026-10-19). Its two passes differ widely on Node (1.05 s then 8.1 s); the 4.58 s figure is the median of the four, and against that spread the comparison counts as no clear change rather than a fork win. +Upstream's supabase sync completes since upstream fixed [colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565) on 2026-10-19. Its spread stays wide (8.17 s median of four this run, against 4.58 s last run), so that comparison still counts as a fork win only at its own wide spread. Two new losing cells appeared in the 2026-10-28 re-measure: **discourse sync** — the fork takes 1.75 s against upstream's 1.19 s on Node and 1.20 s against 0.92 s on Bun, outside spread in every sample — and **gin index** — 1.02 s against 0.96 s on Node and 0.91 s against 0.77 s on Bun, ~0.05–0.14 s of fixed fork overhead on a 119-file index. Both are real and stand as the remaining regression cells; every other index and sync cell is a fork win or a tie. ## The supabase resolution regression @@ -150,7 +150,7 @@ Of 76 comparisons the fork is better in 57, shows no clear change in 10 and is w ## Follow-up merges -Pull requests landed on `fork/consolidated` after the tables above were generated, each measured on its stacked branch build against the same upstream numbers on the same host. The scorecard above reflects the pre-follow-up tree; the current head is `645abbb8`. +Pull requests landed on `fork/consolidated` after the tables above were generated, each measured on its stacked branch build against the same upstream numbers on the same host. The index and sync tables above were regenerated 2026-10-28 at the current head, `de5a7b80`. | PR | Merge commit | Change | Measured result | |---|---|---|---| @@ -169,5 +169,8 @@ Pull requests landed on `fork/consolidated` after the tables above were generate | [#570](https://github.com/bompus/codegraph/pull/570) | `e31b8947` | `bare_call_receiver` memoized on a content hash of ref-identifying fields; `is_generated_file`'s ~25 regexes memoized per unique path; both memos cleared at each batch boundary | pretix exact parity; with #571–#572, settle 4.57 → ~3.5 s (−23%; 3.2–3.7 s across two profiled index runs) | | [#571](https://github.com/bompus/codegraph/pull/571) | `dfb8866f` | `local_declaration_scope` memo keyed by node id in its own map — drops the `format!` alloc the shared map needed for namespacing | parity 8/8; with #570/#572, settle 4.57 → ~3.5 s | | [#572](https://github.com/bompus/codegraph/pull/572) | `645abbb8` | `apply_language_gate` filters in place (`retain`) — drops a `Vec>` alloc per ref | parity 8/8; timing-neutral alloc shave | +| [#573](https://github.com/bompus/codegraph/pull/573) | `77c82438` | This follow-up table through #572 | Documentation only | +| [#574](https://github.com/bompus/codegraph/pull/574) | `a42f5cb8` | `callerTests` batches its per-hop caller lookups (one query per hop instead of one per candidate) | 50 focused tests green; explore-path N+1→2 queries per hop | +| [#575](https://github.com/bompus/codegraph/pull/575) | `de5a7b80` | `daemon-registry` and `mcp-daemon` suites get the 60 s process-lookup allowance `daemon-lock-holder` already had — Windows PowerShell `Get-CimInstance` can exceed the production 5 s default | 43 tests green; three prior Windows lifecycle flakes on unrelated PRs explained, none were code regressions | -What is left on the board after the merges: pretix-Bun index (~+2%, the 0.5 s settle tail plus ~1.4 s of serial post-phases — at the measured optimum of the tuning explored), the pretix and supabase-Bun index-memory cells (the 12 in-process worker heaps that buy the index-time wins), a ~0% gin file-watcher cell inside noise, and the still-diffuse per-candidate gate cost on vendored JS — #569–#572 cut pretix's resolution settle 4.73 → ~3.5 s (−26%, median of two profiled runs) but the residual is spread across many small gates rather than one dominant one. Everything else in the 9-cell loss list is now a win: pretix-Bun sync (#559), n8n busy-MCP memory (#560) and the gin concurrent-explore wave (#560). +What is left on the board at `de5a7b80`, after the 2026-10-28 re-measure above: **gin index** (+6% Node / +18% Bun — ~0.05–0.14 s of fixed fork overhead on a 119-file index, outside spread every sample) and **discourse sync** (+47% Node / +31% Bun — ~0.3–0.6 s, outside spread every sample) are the only remaining loss cells. pretix-Bun index — the former last residual — is now a tie (7.15 s vs 7.20 s). The pretix and supabase index-memory cells remain the disclosed trade (12 in-process worker heaps), though the fork now peaks *lower* than upstream on CPython, discourse-Bun, n8n and gin. The still-diffuse per-candidate gate cost is bounded — #569–#572 cut pretix's resolution settle 4.73 → ~3.5 s (−26%) — and #574 batched the last measured explore-path N+1. Sync wins widened sharply this run (n8n-Node 9.07 → 3.29 s, pretix-Node 1.55 → 0.86 s). From ecc50ce2303af4c291adad531fe33f619e83def7 Mon Sep 17 00:00:00 2001 From: bompus Date: Wed, 28 Oct 2026 07:57:51 -0600 Subject: [PATCH 2/2] docs: regenerate scorecard, heatmap and README card from the 2026-10-28 run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "Every result at a glance" count, loss list, heatmap and README card still carried the earlier matrix: 57/10/9 with pretix-Bun index and sync listed as losses. Regenerated from the same raw run as the tables — adapt-flat-output + render-charts over the 2026-10-28 index-sync rows and the carried 2026-10-27 MCP/CLI data — the scorecard is 57/6/13, adds the gin index and discourse sync losses and the Alamofire index-memory loss, and drops the pretix-Bun cells (now inside spread). README card alt text and prose timings aligned to the generated figures (1.7x index, 3.5x sync, 30% memory; supabase 15.6 s, n8n 49.4 s, pretix +34% Bun). --- README.md | 6 +++--- assets/fork-vs-upstream-performance.svg | 4 ++-- .../fork-vs-upstream-heatmap-2026-10-27.svg | 4 ++-- .../fork-vs-upstream-node-bun-2026-10-27.md | 15 +++++++++------ 4 files changed, 16 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 07e5a2c34..98737fd47 100644 --- a/README.md +++ b/README.md @@ -200,9 +200,9 @@ Call resolution the fork adds: Upstream `main` at `69af19ac` against the fork at `de5a7b80`, each run on Node.js 24.22.1 and on Bun 1.4.4. Index and sync figures were re-measured 2026-10-28 on the same 16-vCPU WSL2 host after the ambient-`declare module` scan fix and the resolve-pipeline changes below; MCP, command-line and explore figures are carried from the earlier 2026-10-27 run at `bc27fdb4` (unchanged by resolution-path changes). Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). -![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 1.5 times faster, syncs an edited file 1.3 times faster, answers a warm question 1.8 times faster and uses 33% less memory while indexing. The fork is better in 57 of 76 comparisons, shows no clear change in 10 and is worse in 9.](assets/fork-vs-upstream-performance.svg?v=8) +![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 1.7 times faster, syncs an edited file 3.5 times faster, answers a warm question 1.8 times faster and uses 30% less memory while indexing. The fork is better in 57 of 76 comparisons, shows no clear change in 6 and is worse in 13.](assets/fork-vs-upstream-performance.svg?v=9) -The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 45% (pretix) to 72% (n8n), with discourse the one loss (+47%) — and on Bun it wins six of seven the same way, discourse again the loss (+31%). Full indexing: the fork wins on n8n (1.7×), CPython (2.1×), discourse (1.6×), pretix-Node (7.48 s against 9.69 s), supabase-Node (15.6 s against 19.6 s) and Alamofire-Node (1.19 s against 1.42 s), and on the Bun arm of Alamofire, pretix (7.15 s against 7.20 s — a tie inside spread), CPython, discourse, supabase and n8n; gin index is the remaining loss (+6% Node / +18% Bun — ~0.05–0.14 s of fixed fork overhead on a 119-file index). The speed costs some memory: indexing peaks higher on pretix (+36% Node / +33% Bun) and supabase (+14% Node / +13% Bun) — the 12 in-process worker heaps are the disclosed trade for the index-time wins — while the fork now peaks lower than upstream on n8n (−30%), CPython (−9%), gin (−33%) and discourse-Bun (−28%). Cells where the fork still trails: gin index on both runtimes, discourse sync on both runtimes, and the pretix and supabase index-memory cells. The post-merge measurements are in the benchmark doc's [follow-up section](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#follow-up-merges). The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. +The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 45% (pretix) to 72% (n8n), with discourse the one loss (+47%) — and on Bun it wins six of seven the same way, discourse again the loss (+31%). Full indexing: the fork wins on n8n (1.7×), CPython (2.1×), discourse (1.6×), pretix-Node (7.48 s against 9.69 s), supabase-Node (15.6 s against 19.6 s) and Alamofire-Node (1.19 s against 1.42 s), and on the Bun arm of Alamofire, pretix (7.15 s against 7.20 s — a tie inside spread), CPython, discourse, supabase and n8n; gin index is the remaining loss (+6% Node / +18% Bun — ~0.05–0.14 s of fixed fork overhead on a 119-file index). The speed costs some memory: indexing peaks higher on pretix (+36% Node / +34% Bun) and supabase (+14% Node / +13% Bun) — the 12 in-process worker heaps are the disclosed trade for the index-time wins — while the fork now peaks lower than upstream on n8n (−30%), CPython (−9%), gin (−33%) and discourse-Bun (−28%). Cells where the fork still trails: gin index on both runtimes, discourse sync on both runtimes, and the pretix and supabase index-memory cells. The post-merge measurements are in the benchmark doc's [follow-up section](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#follow-up-merges). The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. The fork starts its resolver pool at or above 150,000 pending references. Below that, it can start the pool part-way through if at least 20,000 unresolved references remain and the estimated main-thread work is at least 0.4 s. An explicit `CODEGRAPH_PARALLEL_RESOLVE_MIN` replaces both gates. Resolution proceeds in pages of 40,000 references (`CODEGRAPH_RESOLVE_BATCH` overrides), across up to 12 workers (`CODEGRAPH_RESOLVE_WORKERS` sets the count, `0` disables the pool; `CODEGRAPH_NO_PARALLEL_RESOLVE=1` does the same). Against a pre-change baseline build, five tested corpora indexed 11-48% faster at 56-132% more peak memory with identical graphs ([measurements](docs/benchmarks/adaptive-pool-engage-2026-10-15.md)). @@ -220,7 +220,7 @@ A [2026-10-15 mid-size comparison](docs/benchmarks/midsize-sweep-2026-10-15.md) | supabase (React + Next.js + TS, 10,716 files) | 19.6 s, 4.23 GiB | 15.6 s, 4.83 GiB | 14.5 s, 4.63 GiB | | n8n (Vue + TS, 24,434 files) | 83.0 s, 11.00 GiB | 49.4 s, 7.65 GiB | 46.6 s, 8.43 GiB | -The fork's index was slower than upstream's on five of the seven corpora in the first pass of this run — the supabase and n8n regressions traced to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scanned every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic, and the follow-up above lands the rest: a port of upstream's text tokenizer plus a `specifier → files` index, so each import checks only candidate declaration files instead of every `.d.ts`. The re-measured numbers in this table are the result: supabase went from 41.5 s to 15.5 s on Node and n8n from 178 s to 49.5 s — n8n now beats upstream 1.7× and CPython 2.1×. The remaining index losses from that run — pretix +33% and Alamofire +18% on Node, and the Bun arm of supabase and n8n — traced not to the ambient scan but to the resolve loop's batch boundaries: every 5,000-ref page drained the worker pipeline before the next fanned out. This run stacks three fixes for that — 40,000-ref pages with a 12-worker pool cap (same batch-boundary semantics; pool workers resolve against a fixed snapshot), then bounded work-stealing dispatch with earlier batch fan-out and non-blocking worker recycling, then a memoized near-duplicate pass that stopped re-tokenizing bodies it had just signed — and those losses are gone: pretix-Node wins (7.48 s against 9.69 s), the Bun arms of supabase and n8n win, and pretix-Bun is a tie (7.15 s against 7.20 s). Later in the same stack, the deferred-decode path stopped turning off on framework projects: a framework `extract()` hook no longer forces every file onto decoded-object extraction — its output rides the kernel-buffer transport and merges post-decode ([#563](https://github.com/bompus/codegraph/pull/563); pretix went from 0 to 1,416 kernel-path files, n8n-Node index RSS −10%). The benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). The trade is memory: indexing peaks higher on pretix and supabase now, lower on the other five, and its one-file sync is faster on six of seven (below). +The fork's index was slower than upstream's on five of the seven corpora in the first pass of this run — the supabase and n8n regressions traced to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scanned every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic, and the follow-up above lands the rest: a port of upstream's text tokenizer plus a `specifier → files` index, so each import checks only candidate declaration files instead of every `.d.ts`. The re-measured numbers in this table are the result: supabase went from 41.5 s to 15.6 s on Node and n8n from 178 s to 49.4 s — n8n now beats upstream 1.7× and CPython 2.1×. The remaining index losses from that run — pretix +33% and Alamofire +18% on Node, and the Bun arm of supabase and n8n — traced not to the ambient scan but to the resolve loop's batch boundaries: every 5,000-ref page drained the worker pipeline before the next fanned out. This run stacks three fixes for that — 40,000-ref pages with a 12-worker pool cap (same batch-boundary semantics; pool workers resolve against a fixed snapshot), then bounded work-stealing dispatch with earlier batch fan-out and non-blocking worker recycling, then a memoized near-duplicate pass that stopped re-tokenizing bodies it had just signed — and those losses are gone: pretix-Node wins (7.48 s against 9.69 s), the Bun arms of supabase and n8n win, and pretix-Bun is a tie (7.15 s against 7.20 s). Later in the same stack, the deferred-decode path stopped turning off on framework projects: a framework `extract()` hook no longer forces every file onto decoded-object extraction — its output rides the kernel-buffer transport and merges post-decode ([#563](https://github.com/bompus/codegraph/pull/563); pretix went from 0 to 1,416 kernel-path files, n8n-Node index RSS −10%). The benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). The trade is memory: indexing peaks higher on pretix and supabase now, lower on the other five, and its one-file sync is faster on six of seven (below). Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s; [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) put the cause in upstream's code). At `69af19ac` upstream's CPython index is slower than at `986de95c` (53.7 s against 40.2 s on this host) while the fork's keeps improving, so the fork's CPython lead is 2.1× this run. diff --git a/assets/fork-vs-upstream-performance.svg b/assets/fork-vs-upstream-performance.svg index 5b9566a6f..86d5ff0f9 100644 --- a/assets/fork-vs-upstream-performance.svg +++ b/assets/fork-vs-upstream-performance.svg @@ -1,6 +1,6 @@ Fork vs upstream CodeGraph: headline numbers -On n8n the fork indexes in 49.5 s against 74.9 s, syncs one edited file in 9.07 s against 11.6 s, and answers a warm question in 131 ms against 242 ms. Of 76 comparisons it is better in 57, about the same in 10 and worse in 9. +On n8n the fork indexes in 49.4 s against 83.0 s, syncs one edited file in 3.29 s against 11.6 s, and answers a warm question in 131 ms against 242 ms. Of 76 comparisons it is better in 57, about the same in 6 and worse in 13. -Index on n8n 1.5× faster, CPython 2.2× faster; the fork uses 17–33% less memoryNumbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.Index n8n from scratch1.5× faster74.9 s → 49.5 son Bun: 59.4 s → 45.2 sUpdate after editing one file1.3× faster11.6 s → 9.07 son Bun: 8.68 s → 8.10 sAnswer a question (warm)1.8× faster242 ms → 131 mson Bun: 211 ms → 107 msMemory while indexing n8n33% less12.39 GiB → 8.26 GiBon Bun: 9.23 GiB → 8.28 GiBMCP server memory when idle19% less4.40 GiB → 3.58 GiBon Bun: 3.40 GiB → 1.37 GiBStart the command line2.1× faster73 ms → 35 mson Bun: 36 ms → 21 msWhere the fork is not betterIndexing pretix: 7.20 s → 7.49 s on Bun (+4%)Memory indexing pretix: 3.29 GiB → 4.17 GiB on Node (+27%), 2.89 GiB → 3.79 GiB on Bun (+31%)Memory indexing supabase: 4.02 GiB → 4.73 GiB on Bun (+18%)Syncing one file in pretix: 1.13 s → 1.27 s on Bun (+11%)Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%)8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%)File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%)No clear change (inside the comparison's own run-to-run spread): indexing gin on Node; indexing gin on Bun; indexingAlamofire on Node; indexing pretix on Node; memory indexing Alamofire on Node; memory indexing supabase on Node;syncing one file in gin on Bun; syncing one file in supabase on Node; memory busy on n8n (MCP server) on Bun; cPU idleon gin (MCP server) on Bun.Over all 76 comparisons on Node and Bun: better in 57, no clear change in 10, worse in 9. A difference counts only whenit exceeds that comparison's own run-to-run spread.Measured 2026-10-27 · upstream 69af19ac vs fork 0ee0116c · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted +Index on n8n 1.7× faster, CPython 2.1× faster; the fork uses 9–30% less memoryNumbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.Index n8n from scratch1.7× faster83.0 s → 49.4 son Bun: 60.5 s → 46.6 sUpdate after editing one file3.5× faster11.6 s → 3.29 son Bun: 9.17 s → 2.45 sAnswer a question (warm)1.8× faster242 ms → 131 mson Bun: 211 ms → 107 msMemory while indexing n8n30% less11.00 GiB → 7.65 GiBon Bun: 9.54 GiB → 8.43 GiBMCP server memory when idle19% less4.40 GiB → 3.58 GiBon Bun: 3.40 GiB → 1.37 GiBStart the command line2.1× faster73 ms → 35 mson Bun: 36 ms → 21 msWhere the fork is not betterIndexing gin: 0.96 s → 1.02 s on Node (+6%), 0.77 s → 0.91 s on Bun (+18%)Memory indexing Alamofire: 817 MiB → 1.05 GiB on Node (+32%)Memory indexing pretix: 2.92 GiB → 3.97 GiB on Node (+36%), 2.88 GiB → 3.85 GiB on Bun (+34%)Memory indexing supabase: 4.23 GiB → 4.83 GiB on Node (+14%), 4.10 GiB → 4.63 GiB on Bun (+13%)Syncing one file in discourse: 1.19 s → 1.75 s on Node (+47%), 0.92 s → 1.20 s on Bun (+31%)Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%)8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%)File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%)No clear change (inside the comparison's own run-to-run spread): indexing pretix on Bun; memory indexing CPython onNode; memory indexing discourse on Node; syncing one file in gin on Bun; memory busy on n8n (MCP server) on Bun; cPUidle on gin (MCP server) on Bun.Over all 76 comparisons on Node and Bun: better in 57, no clear change in 6, worse in 13. A difference counts only whenit exceeds that comparison's own run-to-run spread.Measured 2026-10-28 · upstream 69af19ac vs fork de5a7b80 · Node 24.21.0, Bun 1.4.3 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted diff --git a/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg index 69099d072..13ea49722 100644 --- a/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg +++ b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg @@ -1,6 +1,6 @@ Fork vs upstream CodeGraph: every result -Heatmap of the fork's change against upstream for full index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. Of 76 comparisons it is better in 57, about the same in 10 and worse in 9. +Heatmap of the fork's change against upstream for full index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. Of 76 comparisons it is better in 57, about the same in 6 and worse in 13. -Every result at a glanceEach cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better beyond the comparison’s own run-to-run spread, amber = inside it, red = worse, grey = inconclusive or no data.Full index: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node+6%+4%−10%−54%−30%−15%−34%Fork on Bun−9%−30%+4%−51%−33%−11%−24%Full index: peak memoryginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−28%+1%+27%−17%−4%+12%−33%Fork on Bun−50%−10%+31%−22%−30%+18%−10%One-file sync: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−17%−40%−4%−55%−50%−40%−22%Fork on Bun0%−42%+11%−58%−45%−71%−7%MCP server on n8nStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−10%−46%−20%−33%+18%−19%−19%Fork on Bun−10%−50%−17%−46%0%−60%−49%MCP server on ginStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−24%−7%+18%−6%−26%−23%−43%Fork on Bun−31%−13%+5%0%−42%−33%−44%Command-line startup on gin--versionstatusexploreFork on Node−52%−32%−47%Fork on Bun−42%−30%−61%Measured 2026-10-27 · upstream 69af19ac vs fork 0ee0116c · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted +Every result at a glanceEach cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better beyond the comparison’s own run-to-run spread, amber = inside it, red = worse, grey = inconclusive or no data.Full index: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node+6%−16%−23%−53%−39%−20%−40%Fork on Bun+18%−27%−1%−51%−38%−14%−23%Full index: peak memoryginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−33%+32%+36%−9%0%+14%−30%Fork on Bun−47%−9%+34%−28%−28%+13%−12%One-file sync: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−23%−38%−44%−56%+47%−63%−72%Fork on Bun−10%−41%−46%−59%+31%−71%−73%MCP server on n8nStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−10%−46%−20%−33%+18%−19%−19%Fork on Bun−10%−50%−17%−46%0%−60%−49%MCP server on ginStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−24%−7%+18%−6%−26%−23%−43%Fork on Bun−31%−13%+5%0%−42%−33%−44%Command-line startup on gin--versionstatusexploreFork on Node−52%−32%−47%Fork on Bun−42%−30%−61%Measured 2026-10-28 · upstream 69af19ac vs fork de5a7b80 · Node 24.21.0, Bun 1.4.3 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted diff --git a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md index 85809726e..7cdcdc25e 100644 --- a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md +++ b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md @@ -136,18 +136,21 @@ The fork-Node server's busy memory on n8n (4.54 GiB against upstream's 3.85) is ## Every result at a glance -![Heatmap of the fork's change against upstream for index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup, on Node and on Bun. Lower is better; cells inside the comparison's own run-to-run spread are marked no clear change.](fork-vs-upstream-heatmap-2026-10-27.svg?v=4) +![Heatmap of the fork's change against upstream for index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup, on Node and on Bun. Lower is better; cells inside the comparison's own run-to-run spread are marked no clear change.](fork-vs-upstream-heatmap-2026-10-27.svg?v=5) -Of 76 comparisons the fork is better in 57, shows no clear change in 10 and is worse in 9. The 9, generated from the same data as the heatmap: +Of 76 comparisons the fork is better in 57, shows no clear change in 6 and is worse in 13. The 13, generated from the same data as the heatmap: -- Indexing pretix: 7.20 s → 7.49 s on Bun (+4%) -- Memory indexing pretix: 3.29 GiB → 4.17 GiB on Node (+27%), 2.89 GiB → 3.79 GiB on Bun (+31%) -- Memory indexing supabase: 4.02 GiB → 4.73 GiB on Bun (+18%) -- Syncing one file in pretix: 1.13 s → 1.27 s on Bun (+11%) +- Indexing gin: 0.96 s → 1.02 s on Node (+6%), 0.77 s → 0.91 s on Bun (+18%) +- Memory indexing Alamofire: 817 MiB → 1.05 GiB on Node (+32%) +- Memory indexing pretix: 2.92 GiB → 3.97 GiB on Node (+36%), 2.88 GiB → 3.85 GiB on Bun (+34%) +- Memory indexing supabase: 4.23 GiB → 4.83 GiB on Node (+14%), 4.10 GiB → 4.63 GiB on Bun (+13%) +- Syncing one file in discourse: 1.19 s → 1.75 s on Node (+47%), 0.92 s → 1.20 s on Bun (+31%) - Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%) - 8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%) - File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%) +No clear change (inside the comparison's own run-to-run spread): indexing pretix on Bun; memory indexing CPython on Node; memory indexing discourse on Node; syncing one file in gin on Bun; memory busy on n8n (MCP server) on Bun; CPU idle on gin (MCP server) on Bun. + ## Follow-up merges Pull requests landed on `fork/consolidated` after the tables above were generated, each measured on its stacked branch build against the same upstream numbers on the same host. The index and sync tables above were regenerated 2026-10-28 at the current head, `de5a7b80`.