Repository navigation
storage: seed load generator offset_committed per export - #39781
Closed
peterdukelarsen wants to merge 1 commit into
Closed
peterdukelarsen wants to merge 1 commit into
peterdukelarsen wants to merge 1 commit into
Conversation
The load generator seeded every export's `offset_committed` with 0 whenever the generator resumed from offset 0, which is the case as soon as one export has never committed, for example a table just added to the source. Every other export then reported 0 until the first committed upper for it arrived, below the value it had reported before, which worker 0 logs as `total gauge offset_committed erroneously regressed`. Seed each export from its own resume upper instead, the same way the PostgreSQL and SQL Server sources do. An export that has never committed still starts at 0, and the generator still replays from the least resume upper. The window in which the 0 seed was reported closes before the first report in practice, since the reclocked committed uppers reach the operator first, so this is a consistency change with no test that demonstrates it. Part of: SS-436 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FTqJbQjM1u9PHMcwj3jRcR
Contributor
Author
|
Didn't reproduce the issue. Spurious fix from Claude |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The load generator seeded every export's
offset_committedwith 0 whenever the generator resumed from offset 0, which is the case as soon as one export has never committed, for example a table just added to the source. Every other export then reported 0 until its first committed upper arrived, below the value it had reported before, which worker 0 logs astotal gauge offset_committed erroneously regressed.Part of: SS-436
Changes
Each export is seeded from its own resume upper instead, the same way the PostgreSQL and SQL Server sources do after #39780. An export that has never committed still starts at 0, and the generator still replays from the least resume upper.
Tests
None demonstrate this change. A cluster workflow that adds a table to an auction source with a 100ms and then a 1ms tick never logged a regression before the change: the reclocked committed uppers reach the operator before its first report, so the 0 seed was overwritten in time in every run. This is a consistency change that closes the window by construction rather than a fix for an observed occurrence. The occurrence attached to SS-436 was a PostgreSQL source, which #39780 addresses.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.🤖 Generated with Claude Code
https://claude.ai/code/session_01FTqJbQjM1u9PHMcwj3jRcR