Add full-text search over build failure logs - #656
Merged
Conversation
Store fail.txt and diff.txt excerpts of failed builds in Postgres so they can be searched with pg_trgm, since S3 offers no full-text search. Excerpts are capped at 100KB per report and captured right after scan_recent_ltsv creates a report; S3 errors are logged and skipped so they never break report ingestion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
log_excerpts:backfill fills excerpts for past failed builds (default last 1 year, FROM/TO to override) and resumes from the max stored report_id when interrupted. log_excerpts:prune deletes excerpts older than KEEP_DAYS for Heroku Scheduler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Search failed builds by excerpt text, revision prefix, server, branch and date range. Reports are narrowed by existing indexes before the trigram match, and results are paginated at 50 rows to stay within the Heroku router timeout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep the revision query parameter for machine clients (a planned MCP server needs to check whether reports exist for a commit), but drop the visible input since it is not useful for humans. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Put the submit button right of the text field and spread the filter row so both rows share the same right edge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The resume point came from the highest already-captured report_id, but scan_recent_ltsv captures new failures as they arrive, so that id is always the newest report and the task skipped everything behind it. It printed "backfilling 0 reports" on every run after the first live capture, which is exactly when the past year still needed importing. Select reports that have no excerpt instead, so an interrupted run resumes and a report whose fetch failed is retried. FORCE=1 refetches reports that already have one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
KEEP_DAYS= or KEEP_DAYS=abc took the empty-string branch and fell to 0 via to_i, which made prune delete every excerpt. FROM and TO parsed in the local zone while reports.datetime is UTC, and an unparseable date raised a bare backtrace. Every option now aborts with a message naming the offending value. Backfill counted loop iterations, so a run where every fetch raised still finished "done: N/N" and exited 0. Count captured, empty and failed separately and exit nonzero when anything failed. Add DRY_RUN to prune, and report progress from both tasks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The two sections were joined and then truncated as a whole, so a fail.txt over the limit left no room for the diff at all. Six of 25 sampled reports hit the cap, meaning a quarter of them were searchable only through their fail output. Give each section an equal share and let one donate its unused share to the other, so the common case where one side is small is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
?page[]=1 made params[:page] an Array, which has no to_i, so the request reached the browser as a 500. A page number past the bigint range produced an OFFSET PostgreSQL rejects, with the same result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The results table reuses table.reports, whose failure colour targets columns 6 through 9. Search has a different column order, so Match and Diff came out red on every row while meaning nothing. Scope the table with its own class and clear those two, leaving Summary red. Also give the Match snippet a width and a title, matching how Summary and Diff already truncate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
chkbuild keeps its failure logs on S3, which offers no full text search. S3 Select is closed to new users, and Athena and CloudWatch sit outside the Heroku stack. This imports the failing excerpt into Postgres and searches it with
pg_trgm, which fits log output better thantsvectorbecause the content is not natural language.log_excerptsstores thefail.txtanddiff.txtof failed builds only, capped at 100KB per report and split between the two so a largefail.txtcannot crowd the diff out. Measured against production that is 11,861 builds over the last year and roughly 270MB before the GIN index. Indexing diffs for successful builds as well would pass 10GB a year, sincedifferent_sectionsis set on almost every build, so those are left out.Report.scan_recent_ltsvcaptures new failures as they arrive, sofetch_recentalready covers ongoing builds and no new scheduled job is needed. A rake task backfills past builds and another drops old excerpts./searchnarrows by server, branch and date range on the existing indexes before the trigram match, and pages at 50 rows to stay inside the Heroku router timeout.revisionworks as a query parameter for machine clients but has no visible field.Development and test run on SQLite, where
pg_trgmdoes not exist, so both the migration and the query branch on the adapter.log_excerpts:pruneneeds to be added to Heroku Scheduler after deploy.