Backfill excerpts for servers registered under http - #657
Merged
Conversation
Backfill selected servers with `uri LIKE 'https://...'` while capture used a regexp accepting either scheme. Production registers 121 of 125 servers under http, so the first backfill covered 1,143 of the 11,861 failed builds in the last year and reported success. Put the predicate on `Server` as both a scope and an instance method so the SQL and Ruby forms stay together, and use it everywhere the question is just whether a server lives in the rubyci S3 bucket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Most server rows carry an http:// URI, so excerpts for them were fetched in the clear. Rewrite the scheme before connecting. The bucket uses S3 Intelligent-Tiering, and an object that has aged into an archive tier answers GET with 403 InvalidObjectState while HEAD still answers 200. That is not retryable without restoring the object, so treat it like a missing one. It accounted for all 26 failures of the first backfill, and sampling puts it near 2 percent of objects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ruby block-buffers stdout when it is not a tty, so a run under `heroku run:detached` showed no progress for its whole duration and then flushed every line at once on exit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first backfill run reported success after covering 1,143 of the 11,861 failed builds in the last year. Backfill picked servers with
uri LIKE 'https://rubyci.s3.amazonaws.com/%', butLogExcerpt.captureused a regexp accepting either scheme, and production registers 121 of its 125 servers underhttp://. Live capture therefore kept working while backfill silently skipped nine tenths of its input.The predicate now lives on
Serveras both a scope and an instance method, so the SQL and Ruby forms sit next to each other instead of drifting apart in three files. Rerunning backfill after this brings the last year from 1,143 reports in scope to 11,862.Since most rows carry an
http://URI, excerpts for them were also fetched without TLS.fetch_textnow rewrites the scheme before connecting.All 26 failures of the first run were one thing. The bucket uses S3 Intelligent-Tiering, and an object that has aged into an archive tier answers GET with
403 InvalidObjectStatewhile HEAD still answers 200, which is why measuring object sizes beforehand did not reveal it. Restoring such an object is neither free nor quick, so this is treated like a missing object and stored as an empty excerpt rather than retried on every run. Sampling 114 objects across five years puts it near 2 percent, with no strong correlation to age.$stdout.syncis set because Ruby block-buffers stdout when it is not a tty, so a run underheroku run:detachedprinted nothing for its whole duration and flushed every progress line at once on exit.