feat(fast-inbox): detect inbox bucket reorgs by L1 block hash and roll back to the last canonical bucket - #25354
Open
spalladino wants to merge 2 commits into
Conversation
…l back to the last canonical bucket The archiver decides it is in sync with the Inbox by comparing message count and consensus rolling hash. An L1 reorg that re-mines the same messages in a different block leaves both unchanged while rewriting the bucket metadata built from that block: its timestamp, sequence number and boundaries. Those feed the censorship cutoff and the bucket hint a proposer publishes, so stale metadata costs a slot. Every sync pass now compares the L1 block hash each unverified bucket was opened in against the live chain, walking up from a canonicality watermark the store keeps. Buckets sharing an opening block cost a single read. On a mismatch the archiver rolls back to the last canonical bucket and rewinds the message syncpoint to just before its L1 block, so the normal fetch re-downloads the reorged blocks; the rolling-hash rollback stays as the fallback. Being at or below the finalized L1 block is not on its own a reason to skip a bucket: a node that was not syncing when its block was reorged out holds an orphaned block that has since been buried below finality. The watermark therefore only advances over a bucket this node has itself compared against a chain that had it finalized, and it is lowered whenever messages are removed.
…mically Review follow-ups to the bucket canonicality check. A bucket keyed on an L1 timestamp can absorb messages from a later block that shares it, and a reorg of that later block alone splits the bucket while its opening block stays canonical and the Inbox's total and rolling hash are unchanged. Bucket snapshots now record the closing L1 block as well as the opening one, both are compared against the live chain, and the watermark only advances once the closing block is finalized. The snapshot type keeps this internal: the check reads bucket spans rather than whole buckets. The check also runs before the "L1 has not moved forward" guard, so a head re-mined at the same number is caught instead of short-circuiting the whole message sync until the chain advances. Rolling back now resolves the target syncpoint before writing, then removes the messages and moves the syncpoint in one transaction, so an interruption cannot leave truncated messages behind a syncpoint that would never fetch them again. The rolling-hash fallback shares that path. Re-delivering a stored bucket from a different L1 block also lowers the watermark, so a bucket is never trusted on a check made against blocks it no longer sits on. Finally, the L1 reads are made in bounded parallel and capped per pass, so an unfinalized tail cannot grow the per-tick read count without limit, and a read that keeps failing is reported at warn rather than debug, since it blocks every bucket above it from ever being verified.
spalladino
force-pushed
the
spl/inbox-bucket-canonicality-check
branch
from
August 29, 2026 11:13
6c29ddd to
4adcb36
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The archiver now checks that the Inbox buckets it holds sit on L1 blocks that are still canonical, and rolls back to the last canonical bucket when one of them does not.
Context
The archiver decides it is in sync with the Inbox by comparing two numbers against L1: the total message count and the consensus rolling hash (
l1_synchronizer.ts,localStateMatches). With the rolling hash covering the messages and nothing else, this reorg is invisible to both:Same messages in the same order, so the same count and the same hash — and the existing rollback cannot help either, since it looks for the last message whose rolling hash still matches and they all do. What did change is the metadata the archiver derives from the L1 block: the bucket's timestamp, and (when the reorg merges or splits blocks) its sequence number and boundaries. A bucket's timestamp is what the censorship cutoff is compared against and the recency key the sequencer's bucket selection walks; its sequence number is the
bucketHintL1 looks the bucket up with, so a stale one gets the proposal reverted withRollup__InvalidInboxRollingHash.#25319 / #25322 / #25327 close this by committing the bucket structure and timestamps into the rolling hash itself, which makes the re-time visible to the check that already exists. This PR is the alternative for keeping the rolling hash over messages only: leave the hash alone and check the bucket metadata directly.
Approach
Each bucket snapshot records the number and hash of the L1 block it was opened in (#25350), and now of the block it was closed in as well. Every sync pass reads those blocks from L1 and compares hashes, before the count-and-hash comparison runs:
canonicalVerifiedThroughSeqwatermark the message store keeps. Buckets at or below it are never looked at again; buckets above it are checked every pass. Rollover siblings share an opening block and so cost a single read.block.timestamp, so where consecutive L1 blocks may share a timestamp (anvil with manual mining) one bucket can absorb messages from several blocks. Reorging only the later one splits the bucket in two while its opening block stays canonical and the message count and rolling hash are untouched — nothing else in the pass would notice. The closing block is store-internal: the check reads bucket spans (InboxBucketL1Span), and theInboxBuckettype consumers see is unchanged.warn, removes every message absorbed after the last canonical bucket, and rewinds the message syncpoint to just before that bucket's opening L1 block — both in a single store transaction, with the target syncpoint resolved beforehand, so an interruption cannot leave truncated messages behind a syncpoint that would never fetch them again. The normal flow then re-downloads the reorged blocks and rebuilds the snapshots with the timestamps and sequence numbers the canonical chain gives them. Since verification runs upward, the last canonical bucket is simply the previous one — no reverse walk."Below finalized" is not a free pass. The existing rollback may stop at the finalized block, because a message at or below it cannot have been reorged while the node was watching. That reasoning does not extend to a node that was not watching: it can hold a block that was orphaned before finalization and has since been buried below it, and no amount of finality will ever make that block canonical. So the watermark records what this node has itself verified, not what the chain has finalized — a bucket sitting below finality that was never checked is still checked once.
RPC cost. One
eth_getBlockByNumberper distinct unverified L1 block per sync pass, issued 4 at a time. In the steady state the watermark trails the finalized block, so that is the blocks within the finality window that actually carried messages, deduplicated; on a quiet chain it is one or two, and it stops entirely once the store has nothing above the watermark. If finality stalls the unverified tail grows without bound, so a pass checks at most 32 buckets and the rest wait for the next one — the watermark is a prefix, so progress is never lost. A read that fails ends the pass there and is logged atwarn(thinned out when it keeps failing on the same bucket), since a block that cannot be read blocks every bucket above it from ever being verified while the sync itself carries on.Tests
FakeL1Statecan now replace a range of L1 blocks with fresh hashes (reorgL1BlocksFrom), re-mine one block's messages into another (retimeMessages, which also covers the merge case), re-mine a block with its messages reordered (reorderMessagesAtL1Block), mine a block carrying an earlier block's timestamp (shareTimestampWithL1Block) and re-mine such a block with a timestamp of its own (splitCoTimestampedL1Block). It also records the L1 blocks read by number and can make those reads fail, so tests assert on reads without reaching into mock call arguments.New
archiver-synccases: a pure re-time (count and rolling hash unchanged, timestamp and block hash updated, exactly one warning); a merge of two buckets into one with message indices preserved; a bucket spanning two co-timestamped blocks split by a reorg of the later one; a reorder rolled back by the bucket walk without reaching the per-message log queries; a head re-mined at the same number; rollover siblings re-downloaded whole when the rollback lands on their shared opening block; the watermark checking a below-finality bucket once and never again; a bucket that is never finalized being re-checked every pass; a spanning bucket waiting on its closing block to finalize; a bucket reorged out across a gap in syncing and detected even though it is now below finality; a failing L1 read warning and verifying nothing; the per-pass bound spreading the walk over several passes; and a catch-up over many buckets from a synced store using bulk events only.message_storegets the span accessors, the span shortening when a removal cuts a bucket, and the watermark being lowered both by a removal and by a re-delivery from another L1 block.Both regressions were confirmed red before the fix: reverting the closing-block check fails the split case, and moving the walk back after the "L1 did not advance" guard fails the same-height re-mine.
Ran: full
yarn build,yarn format,yarn lint archiver stdlib, archiver 597/597 (20 suites), stdlib messaging 57/57. No e2e.Stacked on #25351
Fixes A-1880