Skip to content

fix(server): run migrations on a dedicated connection, decoupled from the query pool's 60s timeout (JEF-590) - #150

Merged
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-590-isolate-the-migrate-connections-timeouts-own-lock_timeout
Jul 28, 2026
Merged

fix(server): run migrations on a dedicated connection, decoupled from the query pool's 60s timeout (JEF-590)#150
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-590-isolate-the-migrate-connections-timeouts-own-lock_timeout

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Closes JEF-590.

Problem

sqlx::migrate! ran on the shared query pool and inherited its 60s
statement_timeout — that killed migration 0017's CREATE INDEX mid-run and
crashlooped the pod (JEF-580). There was no lock_timeout either, so DDL that
can't get its lock would queue ahead of ingest and head-of-line-block the
table for as long as the lock is held.

Fix

db::migrate now runs on a single dedicated PgConnection, independent of
the query pool from connect():

  • lock_timeout=3s — a migration that can't acquire its lock aborts fast
    and fails startup loudly, instead of queuing ahead of ingest.
  • statement_timeout=0 (unbounded), decoupled from the query pool's 60s — a
    legitimate transactional migration on a large table isn't capped at the
    app's query-latency bound. Per ADR 0021, heavy/locking DDL (e.g.
    CREATE INDEX CONCURRENTLY) belongs in the separate online-DDL lane, not
    sqlx::migrate!, so what runs here is expected to stay short regardless.

The query pool's settings in connect() are untouched (still 60s
statement_timeout for app queries).

Implementation choice

Kept migrate's connection settings decoupled from the pool's connect_options()
by re-parsing the URL rather than cloning the pool's options and appending — the
pool's options already carry statement_timeout=60s, and Postgres's -c-style
option string doesn't dedupe by key, so appending would rely on implicit
"last -c wins" ordering. Building fresh options directly from the URL (mirroring
connect()'s own style) is more explicit. This changed migrate's signature from
&PgPool to &str (the database URL); updated the three call sites
(main.rs, tests/smoke.rs, otlp.rs's unit test) accordingly — each already had
the URL in scope, so the diff stayed small.

Testing

Added db::tests::migrate_connection_aborts_fast_on_a_held_lock (skips
cleanly without DATABASE_URL, same convention as the rest of the suite): a
competing connection holds an ACCESS EXCLUSIVE lock on a throwaway table in
an open transaction, then a connection configured exactly like migrate()'s
attempts a trivial ALTER TABLE against that same table — asserts it errors
with a lock_timeout, in well under the test's 10s ceiling, rather than
hanging.

Ran locally against the docker-compose Postgres:

  • cargo fmt — clean
  • cargo check — clean
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo test --locked — 62 lib tests + 66 integration tests pass, including
    the new lock_timeout test

Scope notes

  • docs/adr/0021-online-ddl-lane.md (referenced in the new doc comment) is
    landing on a separate branch (chore/dependabot-automerge-adr-0021) not
    yet merged to main as of this branch's base — the invariant it records
    (heavy/locking DDL → a future online-DDL lane, transactional migrations
    stay short) is exactly what this ticket assumes and documents. No code
    dependency, just a doc reference; safe to merge in either order.
  • Did not touch server/migrations/ per the ticket's instructions.

… the query pool's 60s timeout (JEF-590)

sqlx::migrate! ran on the shared query pool and inherited its 60s
statement_timeout with no lock_timeout at all — that 60s killed migration
0017's CREATE INDEX and crashlooped the pod (JEF-580), and a migration
blocked on a lock would otherwise queue ahead of ingest indefinitely.

db::migrate now opens its own PgConnection with lock_timeout=3s (abort fast
rather than head-of-line-block a table) and statement_timeout=0 (unbounded,
independent of the query pool's 60s bound — heavy/locking DDL belongs in the
separate online-DDL lane per ADR 0021, so a migration run here is expected
to stay short regardless). The query pool from connect() is unchanged.

Adds a connection-level test: a competing transaction holds an ACCESS
EXCLUSIVE lock on a throwaway table, and a trivial DDL statement on a
connection configured like migrate()'s aborts within lock_timeout instead
of hanging.
@thejefflarson
thejefflarson merged commit 5a32799 into main Jul 28, 2026
4 checks passed
@thejefflarson
thejefflarson deleted the thejefflarson/jef-590-isolate-the-migrate-connections-timeouts-own-lock_timeout branch July 28, 2026 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant