storage: seed offset_committed per export for PostgreSQL sources

[peterdukelarsen]

Motivation

The PostgreSQL source seeded every export's offset_committed with the source-wide resumption point: the least resume upper across exports, with the replication slot's confirmed_flush_lsn standing in for an export that has never committed. The slot trails the committed LSN by up to one feedback interval (1s), so whenever a table was added to a source, or a replica restarted while a table's snapshot was pending, every existing export reported a value below the one it had reported last, and kept doing so on every statistics tick until the pending snapshot finished. Worker 0 of the replica logs each of those reports as total gauge offset_committed erroneously regressed, and so does the storage controller after a replica restart, because it seeds its retained statistics from the persisted statistics collection. A snapshot that takes days produces that line for days.

Closes: SS-450

Changes

Each export is now seeded from its own resume upper, the source itself included: under the subsource syntax it is an export with a persist collection and a committed-upper feedback edge, so it reports its own committed upper like a table. That upper is reclocked from the export's persist upper the same way the pipeline reclocks the committed uppers the export reports from, so the seed does not undercut the last reported value. An export that has never committed keeps the slot position as its fallback. The replication start position stays the least resume upper of the table outputs, and the live path clamps each export to its own seed instead of the shared one.

The SQL Server source has the same shape of seed and is left unchanged here; it can follow in its own PR.

Tests

• offset_committed_seeds unit test in src/storage/src/source/postgres/replication.rs covers the per-export seed computation.
• New pg-cdc workflow offset-committed-seed (test/pg-cdc/mzcompose.py and test/pg-cdc/override/offset-committed-seed-part-*.td) adds a table to a table-syntax source and a subsource to a subsource-syntax source, each under an ACCESS EXCLUSIVE lock on its primary key index so the snapshot stays pending, restarts Materialize with the snapshot pending, and asserts that no regression is logged. The statistics views keep the higher value, so the error line in the logs is the only observable. Before this change the workflow fails on 4 of 5 runs, with roughly 120 to 320 regression lines; the race it depends on is between the slot feedback and the DDL. After this change it passes on every run. The subsource-syntax half was added after the QA review below: without seeding the source's own export it fails with 125 lines, all for the source's own statistic.
• The handoff's Docker repro scripts show 150 to 160 regression lines on the DDL path and about 600 on the restart path before this change, and 0 on both after.

Known residual

One rare occurrence was still observed after the fix on the DDL path, on the third add after an add, release, drop cycle: the export's own resume upper came out about 21KB below the value it had last reported, so the seed undercut it. Eleven further cycles with finer sampling did not reproduce it. It is a much smaller and rarer case than the one this change removes, and the cause is still open.

Tips for reviewer

The main change is offset_committed_seeds and its call site in src/storage/src/source/postgres/replication.rs. The workflow follows the same shape.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label.
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FTqJbQjM1u9PHMcwj3jRcR

[def-]

QA LLM Review

1. MEDIUM -- Old-syntax PostgreSQL sources still log offset_committed erroneously regressed for the source's own statistic

src/storage/src/source/postgres/replication.rs:324

The fix seeds only the exports in table_info, and every other statistic falls back to the source-wide resume_lsn. For a source created with the old syntax (FOR TABLES / FOR ALL TABLES / FOR SCHEMAS, the default unless force_source_table_syntax is on), the source itself is a real export that reports its own upper. Adding a subsource, or restarting while one is snapshotting, therefore still undercuts it with the slot position and logs the regression on every statistics tick, from both clusterd and environmentd, until the snapshot finishes.

Details

The comment's premise ("Statistics that belong to no output, those of the source itself") only holds for new-syntax sources. For old syntax the planner inserts the primary id into source_exports with SourceExportDetails::None (src/adapter/src/coord/catalog_implications.rs:1546). So it gets statistics (storage_state.rs:730) and a committed-upper feedback edge (render.rs:262), and it reports offset_committed through uppers.exports like any table. Only table_info skips it (postgres.rs:153), so its seed is min(..) with the slot substituted for the pending table, which trails what it last reported.

Reproduced on the PR's CI image (v26.46.0-dev.0--pr.g1404497c) with the PR's own workflow, switched to CREATE SOURCE ... FOR TABLES (t1) plus ALTER SOURCE ... ADD SUBSOURCE t2. In 5 of 6 runs it logged about 315 lines in the 15 s DDL window and about 700 more in the 30 s after a restart. Every line regressed to exactly that render's replication reader started lsn= value. The same harness with CREATE TABLE ... FROM SOURCE logged 0 lines in 3 of 3 runs. The new workflow only uses the new syntax, so it cannot see this.

Fix: compute the seeds for every export from config.source_resume_uppers, decoded with MzOffset::decode_row, rather than from table_info. Keep resume_lsn computed from the table outputs, because it is the replication start. The unwrap_or(resume_lsn) fallback then only covers ids with no export. sql_server/progress.rs:129 has the same shape (unwrap_or(source_seed)). There the least table seed tracks the primary's own upper more closely, but the same change applies.

[peterdukelarsen]

Confirmed, thanks. Under the subsource syntax the source itself is an export with a persist collection and a committed-upper feedback edge, so it reports its own committed upper, and the first revision still seeded it with the source-wide resumption point. Reproduced with the workflow's second source switched to FOR TABLES (t3) plus ALTER SOURCE ... ADD SUBSOURCE t4 against the first revision's build: 125 regression lines, all for the source's own statistic, each to exactly that render's replication reader started lsn=.

592c283 seeds every export in source_resume_uppers from its own resume upper, for PostgreSQL and SQL Server alike, and keeps the replication start position as the least resume upper of the table outputs. The workflow now uses the subsource syntax for its second source, and passes with 0 lines on the revised build.

🤖 Generated with Claude Code