SS-492 - test: cover PostgreSQL exports progressing during a held snapshot by martykulma · Pull Request #39420 · MaterializeInc/materialize

martykulma · 2026-09-30T20:00:00Z

Motivation

Adds targeted coverage for the concurrent replication behavior in PostgreSQL source.

Description

Adds a pg_snapshot_pause failpoint to the PostgreSQL snapshot operator. return(<table>) holds that table's copy until the failpoint is deactivated. return(<table>:<millis>) holds it for a fixed time, so the snapshot completes in the same dataflow incarnation. Tables are copied in OID order, so tables ordered before the held one finish their snapshots.

Two new test/cluster workflows share setup and add-tables phases in test/cluster/pg-snapshot-freshness/.

pg-snapshot-freshness holds t_paused indefinitely. It checks that an already hydrated table keeps ingesting CDC and that t_alongside, added in the same transaction, completes its own snapshot. It also checks that CDC for the held table is staged without committing. It then restarts clusterd without the failpoint and checks that every table converges, with the row inserted during the hold appearing exactly once.

pg-snapshot-freshness-release holds t_paused for 30 seconds while CDC for it arrives across several timestamps. It checks that when the hold lifts, the snapshot and all staged CDC commit exactly once without a restart, and that later CDC lands as for any hydrated table.

Verification

Both workflows are new in this PR and rely on the mzcompose default that sets storage_persist_sink_description_lookahead, which is what turns concurrent replication on.

🤖 Generated with Claude Code

linear-code · 2026-10-02T13:02:56Z

SS-492

def- · 2026-10-09T03:15:15Z

QA LLM Review

1. MEDIUM -- Release workflow passes when the held snapshot is re-run after a dataflow restart

test/cluster/pg-snapshot-freshness/04-release.td:57

pg-snapshot-freshness-release says it checks that the snapshot plus the staged CDC commits in the same incarnation, "not a re-run snapshot". It never checks that, and a restart is silently retried into a pass. If the in-place release path errors and the ingestion dataflow restarts, the failpoint holds t_paused again, a fresh snapshot copies all six rows, and every assertion still matches.

Details

The pass-through path in today's code:

Any persist sink error, for example the invalid upper bail at src/storage/src/render/persist_sink.rs:1692, maps to HealthStatusUpdate::halting (src/storage/src/render.rs:416). That triggers a suspend-and-restart after the 5s storage_suspend_and_restart_delay, logged only at info/warn. ci_annotate_errors does not match either line, unlike panics and halting process:.
held_since resets in each incarnation and FAILPOINTS is process-wide, so the restarted dataflow holds t_paused for another 30s. Then the snapshot re-runs at a new LSN that already contains 300-303.
In CI, the SELECT * FROM t_paused step finished 16s in. A restart adds roughly 5s + 30s, well inside the 120s default_timeout set at test/cluster/mzcompose.py:1566. The snapshot_committed and post-release CDC checks pass the same way.

So a bug in exactly the path this workflow exists for, committing staged CDC once the held frontier lifts, surfaces as one restart followed by a green run. To make the claim checkable, assert something only the in-place path produces. For example, require that snapshot_records_staged for t_paused is 2 in the same check as snapshot_committed = true. A re-run snapshot would report 6. Alternatively, assert that mz_internal.mz_source_status_history has no stalled entry for the source during the run.

martykulma · 2026-10-09T03:56:40Z

MEDIUM -- Release workflow passes when the held snapshot is re-run after a dataflow restart

confirmed and fixed

peterdukelarsen

Human-only-review... Looks to me like it tests the key functionality of letting CDC continue during snapshot of a new table.

peterdukelarsen · 2026-10-13T16:59:27Z

+                    // deactivated. `return(<table name>:<millis>)` holds it for that long
+                    // instead, so the snapshot completes in the same dataflow incarnation.
+                    // Tables are snapshotted in OID order, so tables ordered before the paused
+                    // one complete their snapshots.


nit: can we tuck this away in a function?

peterdukelarsen · 2026-10-13T17:15:31Z

+        Testdrive(no_reset=True, default_timeout="120s"),
+        Clusterd(
+            name="clusterd1",
+            environment_extra=["FAILPOINTS=pg_snapshot_pause=return(t_paused:30000)"],


Why did you choose 30s?

guess - making it too long just makes the test take a long time, making it short would cause flake.

peterdukelarsen · 2026-10-13T17:29:01Z

Not going to share anything from review with Claude-code. It of course found some issues, but nothing blocking IMO

Add a `pg_snapshot_pause` failpoint that holds one table's snapshot copy, and two test/cluster workflows built on it. `pg-snapshot-freshness` checks that a hydrated table keeps ingesting CDC while another table's snapshot is held, that a table added alongside completes its own snapshot, and that everything converges after a restart re-runs the snapshot. `pg-snapshot-freshness-release` lifts a timed hold within the same dataflow and checks that the snapshot and the CDC staged behind it land exactly once. Co-Authored-By: Claude <noreply@anthropic.com>

…pshot (#39419) ### Motivation Today a PostgreSQL source snapshots all of its tables before it reads any of the replication stream. This PR lets the replication operator read the stream while the snapshot runs. #39479 then lets each export's frontier advance on its own. ### Description All new behavior is keyed off `storage_persist_sink_description_lookahead` being nonzero. That dyncfg already controls whether the persist sink groups a snapshotting export's updates into one batch, and it defaults to zero in production and to 5s in mzcompose. Concurrent replication is what makes a snapshot accumulate many timestamps, and reading the stream concurrently without the grouping keeps one batch builder per timestamp resident until each reaches the blob target size. The two are therefore one setting rather than a separate flag. With a nonzero lookahead, the snapshot operator emits its rewind requests as soon as the snapshot LSN is known, before any table is copied. The replication operator then reads the stream while tables are copied. Every export still advances together, so replication data is staged in the dataflow until the snapshot completes. This trades cluster memory during hydration for progress on the replication stream. With the lookahead at zero, rewinds are emitted after the copy completes, as before. The snapshot operator reads the setting once, since the early and the late emission must not both happen or both be skipped. Only the snapshot operator reads it in this PR, so a change takes effect for snapshots that start afterwards. parallel-workload already rolls the lookahead through zero and nonzero values. The persist sink logs the resolved lookahead per export at dataflow build time, so a hydration's logs show whether grouping and concurrent replication are on. ### Verification The mzcompose default sets the lookahead to 5s, so the existing mzcompose-based PostgreSQL source tests exercise the early rewind path. #39420 adds targeted coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

…g a snapshot (#39479) ### Motivation Every export's frontier stays at the minimum until the last table's COPY finishes. Adding one large table to a source therefore still stalls ingestion for every table already hydrated in that source. ### Description With `storage_persist_sink_description_lookahead` nonzero, each export's port holds the minimum capability only while that export still has snapshot data to emit or a pending rewind. - Exports that are not being snapshotted drop their snapshot capabilities up front, so they advance with the replication stream. - A snapshotted export releases its snapshot capability as soon as its own copy finishes on each worker. - The replication operator downgrades every port without a pending rewind at each yield point. With the lookahead at zero, all ports advance together, so behavior matches #39419. The replication operator now reads the setting as well. The two operators read it independently at startup, and parallel-workload rolls the lookahead through zero and nonzero values, so a change can land between those two reads and leave the operators in different modes within one dataflow. That is benign in both directions. With only the snapshot operator concurrent, the replication operator holds every port until the rewinds resolve and then releases them all. With only the replication operator concurrent, it still waits for the rewind input to close, which the serial snapshot operator does after the copy. ### Verification The mzcompose default sets the lookahead, so the existing mzcompose-based PostgreSQL source tests exercise per-export progress. #39420 adds targeted coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

martykulma added this pull request to stack #39421 September 30, 2026 20:00

martykulma mentioned this pull request Sep 30, 2026

SS-457 - storage: read the PostgreSQL replication stream during a snapshot #39419

Merged

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from 1ebea0d to d10417a Compare October 1, 2026 15:52

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from d10417a to 9f4ad43 Compare October 2, 2026 11:36

martykulma removed this pull request from stack #39421 October 2, 2026 11:39

martykulma changed the base branch from maz-pg-concurrent-replication to maz-pg-per-export-progress October 2, 2026 11:39

martykulma added this pull request to stack #39480 October 2, 2026 11:39

martykulma mentioned this pull request Oct 2, 2026

SS-492 - storage: let PostgreSQL exports progress independently during a snapshot #39479

Merged

martykulma changed the title ~~test: cover PostgreSQL exports progressing during a held snapshot~~ SS-492 - test: cover PostgreSQL exports progressing during a held snapshot Oct 2, 2026

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from 9f4ad43 to 7e0ed94 Compare October 7, 2026 00:38

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from 7e0ed94 to a1eba8f Compare October 7, 2026 13:45

martykulma marked this pull request as ready for review October 7, 2026 14:20

martykulma requested a review from a team as a code owner October 7, 2026 14:20

martykulma requested a review from peterdukelarsen October 7, 2026 14:26

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from a1eba8f to 8a9055a Compare October 7, 2026 17:01

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from 8a9055a to a4fe7e8 Compare October 7, 2026 17:15

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from a4fe7e8 to 4ae0bb5 Compare October 9, 2026 02:37

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from 4ae0bb5 to 3be8385 Compare October 9, 2026 03:14

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from 3be8385 to ce752f0 Compare October 9, 2026 03:27

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from ce752f0 to aab90be Compare October 9, 2026 03:29

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from fd19975 to 2279d48 Compare October 9, 2026 14:51

peterdukelarsen reviewed Oct 13, 2026

View reviewed changes

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from 2279d48 to a11ee92 Compare October 14, 2026 13:46

peterdukelarsen approved these changes Oct 14, 2026

View reviewed changes

martykulma and others added 3 commits October 14, 2026 11:06

PR comment: validate snapshot row count

a2e7429

PR comment: extract failpoint loop to own function

18c0c44

martykulma force-pushed the maz-pg-snapshot-freshness-tests branch from a11ee92 to 18c0c44 Compare October 14, 2026 15:06

martykulma merged commit 1c6fa09 into main Oct 14, 2026
83 checks passed

martykulma deleted the maz-pg-snapshot-freshness-tests branch October 14, 2026 16:40

Navigation Menu

Repository navigation

SS-492 - test: cover PostgreSQL exports progressing during a held snapshot - #39420

SS-492 - test: cover PostgreSQL exports progressing during a held snapshot#39420
martykulma merged 3 commits into
maz-pg-per-export-progressfrom
maz-pg-snapshot-freshness-tests

martykulma commented Sep 30, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Oct 2, 2026

Uh oh!

def- commented Oct 9, 2026

Uh oh!

martykulma commented Oct 9, 2026

Uh oh!

peterdukelarsen left a comment

Uh oh!

peterdukelarsen Oct 13, 2026

Uh oh!

martykulma Oct 14, 2026

Uh oh!

peterdukelarsen Oct 13, 2026

Uh oh!

martykulma Oct 14, 2026

Uh oh!

peterdukelarsen commented Oct 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

martykulma commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Verification

Uh oh!

linear-code Bot commented Oct 2, 2026

Uh oh!

def- commented Oct 9, 2026

QA LLM Review

1. MEDIUM -- Release workflow passes when the held snapshot is re-run after a dataflow restart

Uh oh!

martykulma commented Oct 9, 2026

Uh oh!

peterdukelarsen left a comment

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

martykulma Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

martykulma Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

peterdukelarsen commented Oct 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

martykulma commented Sep 30, 2026 •

edited

Loading