Repository navigation
perf(markdown): skip link nodes for targets no file can supply - #612
Merged
Merged
Conversation
A Markdown link or path mention with a relative path created an import node and a reference even when no indexed file had that basename, so the reference could never resolve. On supabase those dead targets were 10,120 of 14,038 Markdown import nodes and 10,321 of 11,368 failed Markdown refs. The extractor now drops a slash-shaped target whose basename no indexed file has. Bare names stay: a root-absolute link (`/pricing`), a root-level mention (`Next.js`) or a parent reference can resolve to a route, module or table node by name, which the resolver reaches without a file node. The known basenames come from the scan plus existing DB rows, ride to each parse worker once per spawn on the grammar handshake, and also serve the in-process fallback. Single-file re-index paths keep their old permissive behaviour. Resolved Markdown edges are byte-identical on supabase, n8n, pretix and discourse. Supabase: 152,274 -> 142,154 nodes and 328,137 -> 318,017 edges (dead import nodes and their contains edges), failed Markdown refs 11,368 -> 1,047. Full index median 16.126 s -> 15.920 s (Hyperfine 2.0.0, 8 runs per build across two ABBA rounds, interquartile ranges not overlapping). The payroll-go golden dump loses only the four dead README refs. Full suite 958 files / 11,350 passed.
![bompus-ai-review[bot]](https://avatars.githubusercontent.com/u/253635?s=60&v=4){kind=small}
Review fixes for the dead-target pruning: - Full index only. indexFile() passes no context, so incremental sync keeps every target and a file added later still resolves from the parked reference (colbymchenry#1240). - Keep root-absolute targets and extensionless paths: the resolver can match them to routes, modules or table rows by name, and a root-absolute link is not a relative project file. Only a relative, file-shaped target whose basename no indexed file has is dropped. - Keep every target in the file digest. digestReferences collects the same list unresolvedReferences had, including skipped targets, so a link to an unindexed asset still makes its file findable. Supabase: 152,274 -> 148,652 nodes and 328,137 -> 324,515 edges, 14,038 -> 10,416 Markdown import nodes, 11,368 -> 7,545 failed Markdown refs. Resolved Markdown edges and file digests are byte-identical on supabase, n8n, pretix and discourse. Index time shows no clear change (16.144 vs 16.032 s median, ranges overlap). Full suite 959 files / 11,352 passed.
bompus
changed the title
Review round 2 changed the pruning from the reference to the decorative import node: - Every reference is kept. The resolver's route/module/table arms still see it, and sync can resolve a target added later from the parked failed reference. A sync integration test starts with a full index and adds the target afterwards. - Only the import node and its contains edge are skipped on a full index, when the target's basename no indexed file has. A relative route with a dotted final segment (`api/users.json`) still resolves from the owner. - Skipped targets go into a separate `links:` digest part (64 targets / 800 chars), so a link past the 12-item display digest stays searchable; the test uses a 13th link. Supabase: 152,274 -> 141,880 nodes and 328,137 -> 317,743 edges, import nodes 14,038 -> 3,644, failed Markdown refs unchanged at 11,368. Resolved Markdown edges are byte-identical on supabase, n8n, pretix and discourse. Full index median 16.176 -> 15.953 s (Hyperfine 2.0.0, 8 runs per build, two ABBA rounds, IQRs not overlapping). Full suite 960 files / 11,353 passed.
bompus
changed the title
bompus
added a commit
that referenced
this pull request
Nov 1, 2026
A Markdown table row, list item or fenced command minted a constant or function node plus a contains edge even though the text already lived in the section docstring. On supabase that was 6,222 of 27,380 Markdown nodes. Rows/items now contribute their references to the owning section, commands keep their script calls reference from the section, and each section gets a short commands: summary. Text the 600-character section preview drops stays searchable in a separate text: part of the file digest; every recognized command is kept there even after the 8-command display summary fills. Unlabeled fences only contribute a command when the line looks like one, and now recognize every interpreter extractScriptPath supports (node, powershell, ...), so output, trees and JSON fragments stop becoming commands while Node.js examples keep their script edge. Supabase vs the merged #612 base: 141,880 -> 135,658 nodes, 317,743 -> 311,496 edges, full index median 15.920 -> 15.753 s (Hyperfine 2.0.0, 8 runs per build, two ABBA rounds, IQRs not overlapping). n8n drops 3,719 nodes and discourse 869. References that had resolved to a removed row/command node now fail (25 supabase, 3 n8n, 1 discourse); references from a row/command move to its section. README, the languages reference and the MCP instructions now describe the section-text model. Full suite 960 files / 11,359 passed.
bompus
added a commit
that referenced
this pull request
Nov 1, 2026
A Markdown table row, list item or fenced command minted a constant or function node plus a contains edge even though the text already lived in the section docstring. On supabase that was 6,222 of 27,380 Markdown nodes. Rows/items now contribute their references to the owning section and commands keep their script calls reference from the section. Text the 600-character preview drops stays searchable on the owning node's new search_text column, indexed in nodes_fts but not displayed: late rows, table headers, every recognized command past the 8-command summary, and unrecognized fence lines. Token-boundary dedupe keeps a later CG-713 from being treated as a substring of XCG-713. Unlabeled fences recognize every interpreter extractScriptPath supports (node, powershell, ...). Supabase vs the merged #612 base: 141,880 -> 135,658 nodes, 317,743 -> 311,496 edges; 25 references that pointed at a removed row/command node now fail and 112 re-source to their section. Index time shows no clear change (15.969 vs 16.047 s median, two ABBA rounds, ranges overlapping). Schema v24 adds nodes.search_text and indexes it in nodes_fts; the golden dumps re-baseline only for nodes that carry the field. Full suite 960 files / 11,359 passed.
bompus
added a commit
that referenced
this pull request
Nov 1, 2026
A Markdown table row, list item or fenced command minted a constant or function node plus a contains edge even though the text already lived in the section docstring. On supabase that was 6,222 of 27,380 Markdown nodes. Rows/items now contribute their references to the owning section and commands keep their script calls reference from the section. Text the 600-character preview drops stays searchable on the owning node's new search_text column, indexed in nodes_fts but not displayed: late rows, table headers, every recognized command past the 8-command summary, and unrecognized fence lines. Token-boundary dedupe keeps a later CG-713 from being treated as a substring of XCG-713. Unlabeled fences recognize every interpreter extractScriptPath supports (node, powershell, ...). Supabase vs the merged #612 base: 141,880 -> 135,658 nodes, 317,743 -> 311,496 edges; 25 references that pointed at a removed row/command node now fail and 112 re-source to their section. Index time shows no clear change (15.969 vs 16.047 s median, two ABBA rounds, ranges overlapping). Schema v24 adds nodes.search_text and indexes it in nodes_fts; the golden dumps re-baseline only for nodes that carry the field. Full suite 960 files / 11,359 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Markdown link or path mention whose target basename no indexed file has still minted an import node and a
containsedge, even though the resolver's file-path arm could not match it. The first draft removed the reference too; review showed that loses a Django route link, a target added later by sync, and a link target past the digest budget. The shipped change removes only the decorative node and keeps every reference.Change
codegraph synccan resolve a target added later from the parked failed reference (Incremental sync misses cross-file relations after a new export is added colbymchenry/codegraph#1240).indexFilepasses no context), so a file added later resolves even if the Markdown owner is unchanged.links:part of the file digest (up to 64 targets / 800 characters), so a link past the 12-item display digest is still searchable.grammarhandshake.__tests__/markdown-path-refs.test.tscovers references kept, nodes skipped on a full index, nodes kept on sync, route/basename/symbol/anchor/command retention, a dotted relative route, and a 13th link inlinks:.__tests__/integration/full-pipeline.test.tsstarts with a full index and adds the missing target through sync, then checks the edge.Results
Supabase, full index, base
47839cc4vs this branch:Resolved Markdown
imports/references/callsedge multisets are byte-identical (SHA-256) on supabase (3,058), n8n (2,231), pretix (1) and discourse (624); the only removed rows are the skipped import nodes and theircontainsedges. n8n -419 nodes, discourse -376, pretix -1; failed Markdown reference counts unchanged on all four.Index time: supabase median 16.176 s to 15.953 s (Hyperfine 2.0.0, 8 runs per build across two ABBA rounds, 0.223 s / 1.4%, interquartile ranges 16.170-16.235 vs 15.867-15.982 not overlapping).
Checks
payroll-go.dumploses the.pb.goimport node and its contains edge; its four references are restored (refs 595 -> 599) and the digest gains thelinks:part.Re-index Markdown-heavy projects after upgrading.