fix(server): run migrations on a dedicated connection, decoupled from the query pool's 60s timeout (JEF-590) - #150
Merged
thejefflarson merged 1 commit intoJul 28, 2026
Conversation
… the query pool's 60s timeout (JEF-590) sqlx::migrate! ran on the shared query pool and inherited its 60s statement_timeout with no lock_timeout at all — that 60s killed migration 0017's CREATE INDEX and crashlooped the pod (JEF-580), and a migration blocked on a lock would otherwise queue ahead of ingest indefinitely. db::migrate now opens its own PgConnection with lock_timeout=3s (abort fast rather than head-of-line-block a table) and statement_timeout=0 (unbounded, independent of the query pool's 60s bound — heavy/locking DDL belongs in the separate online-DDL lane per ADR 0021, so a migration run here is expected to stay short regardless). The query pool from connect() is unchanged. Adds a connection-level test: a competing transaction holds an ACCESS EXCLUSIVE lock on a throwaway table, and a trivial DDL statement on a connection configured like migrate()'s aborts within lock_timeout instead of hanging.
thejefflarson
deleted the
thejefflarson/jef-590-isolate-the-migrate-connections-timeouts-own-lock_timeout
branch
July 28, 2026 03:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes JEF-590.
Problem
sqlx::migrate!ran on the shared query pool and inherited its 60sstatement_timeout— that killed migration 0017'sCREATE INDEXmid-run andcrashlooped the pod (JEF-580). There was no
lock_timeouteither, so DDL thatcan't get its lock would queue ahead of ingest and head-of-line-block the
table for as long as the lock is held.
Fix
db::migratenow runs on a single dedicatedPgConnection, independent ofthe query pool from
connect():lock_timeout=3s— a migration that can't acquire its lock aborts fastand fails startup loudly, instead of queuing ahead of ingest.
statement_timeout=0(unbounded), decoupled from the query pool's 60s — alegitimate transactional migration on a large table isn't capped at the
app's query-latency bound. Per ADR 0021, heavy/locking DDL (e.g.
CREATE INDEX CONCURRENTLY) belongs in the separate online-DDL lane, notsqlx::migrate!, so what runs here is expected to stay short regardless.The query pool's settings in
connect()are untouched (still 60sstatement_timeoutfor app queries).Implementation choice
Kept
migrate's connection settings decoupled from the pool'sconnect_options()by re-parsing the URL rather than cloning the pool's options and appending — the
pool's options already carry
statement_timeout=60s, and Postgres's-c-styleoption string doesn't dedupe by key, so appending would rely on implicit
"last
-cwins" ordering. Building fresh options directly from the URL (mirroringconnect()'s own style) is more explicit. This changedmigrate's signature from&PgPoolto&str(the database URL); updated the three call sites(
main.rs,tests/smoke.rs,otlp.rs's unit test) accordingly — each already hadthe URL in scope, so the diff stayed small.
Testing
Added
db::tests::migrate_connection_aborts_fast_on_a_held_lock(skipscleanly without
DATABASE_URL, same convention as the rest of the suite): acompeting connection holds an
ACCESS EXCLUSIVElock on a throwaway table inan open transaction, then a connection configured exactly like
migrate()'sattempts a trivial
ALTER TABLEagainst that same table — asserts it errorswith a lock_timeout, in well under the test's 10s ceiling, rather than
hanging.
Ran locally against the docker-compose Postgres:
cargo fmt— cleancargo check— cleancargo clippy --all-targets -- -D warnings— cleancargo test --locked— 62 lib tests + 66 integration tests pass, includingthe new lock_timeout test
Scope notes
docs/adr/0021-online-ddl-lane.md(referenced in the new doc comment) islanding on a separate branch (
chore/dependabot-automerge-adr-0021) notyet merged to
mainas of this branch's base — the invariant it records(heavy/locking DDL → a future online-DDL lane, transactional migrations
stay short) is exactly what this ticket assumes and documents. No code
dependency, just a doc reference; safe to merge in either order.
server/migrations/per the ticket's instructions.