Repository navigation
perf(index): skip the duplicate edge-index rebuild in the parse window - #611
Merged
Merged
Conversation
A fresh full index drops the edge indexes in beginBulkParseLoad, then endBulkParseLoad rebuilt them, and the batched resolver immediately dropped them again for its own bulk-edge window. When the resolver's window will engage (pending refs >= minRefsForBulkLoad), defer the edge rebuild to the resolver's endBulkEdgeLoad. Small indexes that stay below the window rebuild in the parse phase as before. n8n: parse-index-rebuild 4.47 -> 4.02 s; the resolver's edge recreate still runs; total 52.64 -> 51.27 s across two runs per arm. Edge index count (5) and edge rows (1,236,342) are unchanged.
![bompus-ai-review[bot]](https://avatars.githubusercontent.com/u/253635?s=60&v=4){kind=small}
Review of #611: the parse finally deferred the edge rebuild whenever the pending count cleared the bulk threshold, even when extraction returned success: false and the resolver (the only other rebuilder) was skipped. Defer only after successful parsing, and track the deferral through the outer finally so an aborted or short-circuited run rebuilds the indexes. Rebuilding after the resolver already did is a CREATE IF NOT EXISTS no-op. Test pins the defer/restore pair.
Review of #611: a failed abort-path rebuild was swallowed, and the existing connection's off-thread recoverer did not know the indexes were missing. Register them with deferIndexRecovery() and run recoverIndexesOffThread() (its own connection, 30s busy timeout); a failed recovery leaves them registered for the next open and logs the failure. Test covers the deferred-register/recover path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
beginBulkParseLoad;endBulkParseLoadrebuilt them, and the batched resolver immediately dropped them again for its own bulk-edge window.endBulkParseLoad(deferEdgeIndexes)now letsindexAllskip that first rebuild when the resolver's window will engage (pending refs >= minRefsForBulkLoad). The resolver'sendBulkEdgeLoadrebuilds them after resolution. A run below the window rebuilds in the parse phase as before.Measured (n8n, 2 runs per arm, admission passed)
Tests
npx vitest run: 958 files, 11,345 passed, 46 skipped, 0 failed.README impact
The README measured-results table predates this change; n8n's index figure will move. Re-measuring the corpora is heavy and is recorded as a follow-up rather than hand-editing one row.