Skip to content

feat(fast-inbox): prune the proposed chain when an L1 reorg orphans consumed inbox messages - #25361

Draft
spalladino wants to merge 11 commits into
spl/inbox-consumption-modefrom
spl/inbox-reorg-prune-proposed-chain
Draft

feat(fast-inbox): prune the proposed chain when an L1 reorg orphans consumed inbox messages#25361
spalladino wants to merge 11 commits into
spl/inbox-consumption-modefrom
spl/inbox-reorg-prune-proposed-chain

Conversation

@spalladino

@spalladino spalladino commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Closes the last gap in the node's recovery from an L1 reorg that orphans Inbox messages a block already consumed: the locally proposed chain built on those messages.

The problem

The archiver already rolls back orphaned messages and bucket snapshots (#25354), and the proposer already re-resolves the bucket it consumed before signing and before publishing (#25355). Neither touches the blocks this node already built. Under pipelining those blocks — and the checkpoint that covers them — live in the archiver as proposed state, so after the rollback the node still holds a pending chain whose L1-to-L2 tree contains leaves that no longer exist on L1. Recovery was indirect: the propose reverts, the slot ends without a checkpoint, and the end-of-slot prune throws the blocks away one slot later. In the meantime the proposer of the next slot resolves the parent's bucket by cumulative message total, and after a re-sync a different bucket can carry the same total, so it can build a child on the wrong L1-to-L2 tree.

The rule

A block's L1-to-L2 tree leaf count is the cumulative count of messages it consumed through — that identity is what the whole streaming design rests on. So when the rollback removes messages from index f on, a proposed block whose leaf count is > f consumed at least one removed message; the first such block starts the invalid suffix, and everything from it on goes.

f is the index the rollback rewound to, not the first leaf whose value actually changed, so a reorg that re-mines the same messages in a different L1 block also prunes blocks whose trees are in fact unchanged. That over-pruning is deliberate rather than incidental: the bucket those blocks reference was re-timed and re-numbered by the reorg, so L1 would reject a checkpoint carrying it anyway.

The message rewind and the block prune commit in a single store transaction. Splitting them would make the prune unrecoverable — a crash in between leaves a proposed chain built on messages the store no longer holds, and the next pass re-downloads the canonical messages, finds the local state consistent with L1, and never comes back to those blocks.

The prune runs inside the message step of the sync pass, so it fires before the checkpoint step and every consumer — world state through the tips, the sequencer and validators through their pull checks — sees a consistent local view after a single poll. It is idempotent: a second rollback at the same index finds nothing left to prune.

Two boundaries the predicate gets right by construction: a block whose leaf count equals f consumed only indices below it and is kept, and a block that consumed no messages of its own is pruned only when its parent is.

Published checkpoints are never touched by this path. If the message store disagrees with a checkpoint L1 accepted, one of the two views of L1 is mid-reorg and this is not necessarily the right one — L1 validates the rolling hash in its own chain, so the archive comparison in the checkpoint step stays the authority on published state. Pruning it on a message signal alone would let a flaky RPC unwind real chain. For the same reason nothing is pruned when f falls below the checkpointed tip's leaf count: there every proposed descendant satisfies the predicate, including ones that consumed nothing removed, and acting on it would drop them while the published parent that did consume those messages stays.

The pruned blocks go out on the existing L2PruneUncheckpointed event and are counted under a new inbox_rollback prune-type label; world state follows the tips moving backwards, as it does for every other prune.

Validators

The bucket check resolves the reference locally and compares rolling hashes, so once a validator's own archiver has rewound, a proposal on the orphaned bucket is rejected with bucket_hash_mismatch — including after the single forced re-sync the handler does on a mismatch. Added a test that pins exactly that, driven off a real message store so the rollback, the bucket resolution and the post-sync re-read are the production ones.

One validator-side fix: checkpoint proposal validation caches its verdict by signed-payload hash, and p2p makes two calls for one proposal (the all-nodes validation, then the attestation). A rollback in between prunes the blocks a valid verdict was based on, and the attestation would still be signed off the cached result. A valid cached verdict is now only reused while the checkpoint's last block is still local.

Known gap

A block whose world-state fork was taken before the rollback can still be pushed into the archiver after the prune scanned it. The sequencer forks once per slot and pushes each block it builds as it goes; the validator has the analogous window after re-execution. Such a block lands on the parent that survived the prune, its messages are canonical again by then, and no later prune fires. Closing it exactly needs the builder to carry the message-store state it built against so the insert can reject a stale one; a plain "a proposed block may not consume more messages than the store holds" assertion is correct in production but is violated by the random block fixtures the archiver test suites are built on, so it is not a change that belongs here. Marked with a TODO at the insert site. In practice the block still cannot be published — the proposer re-resolves the bucket by rolling hash before publishing and abandons the slot when it no longer resolves — so the exposure is a stale proposed tip that the end-of-slot prune clears, which is where this stood before this PR.

Tests

  • archiver-sync: a rollback prunes from the first proposed block past the removed index and keeps the ones at or below it (boundary and message-free-successor cases included), evicts the proposed checkpoint covering them, and moves the tips world state follows; a reorg that refills the same indices with different leaves still prunes, since the block consumed the old leaves; a prune that fails leaves the messages in place, so the next pass retries both halves together; checkpointed blocks survive a rollback of the messages they consumed; and a rollback reaching below the checkpointed tip prunes nothing.
  • proposal_handler: the validator rejection described above, and the cached-verdict re-validation.
  • A single-node e2e (single-node/cross-chain/streaming_inbox_reorg) that withholds the checkpoint's propose tx so the consuming blocks are still proposed, checks the consuming block sits above the checkpointed tip before reorging, reorgs the message out and mines a replacement in its place, and asserts the prune event covers the consuming block within a fraction of a slot, published state was not unwound, the orphaned message is not consumable, the replacement is, and a checkpoint built after the reorg reaches L1. This e2e has not been run — it compiles and is wired into CI by the category glob, but it was written without a local run.

No L1 change, no configuration change.

Fixes A-1389

Stacked on #25355

@spalladino spalladino added the ci-draft Run CI on draft PRs. label Aug 29, 2026
…n one transaction

Splitting the two left the prune unrecoverable: a crash or a failure after the messages were dropped left the
proposed chain built on messages the store no longer held, and the next sync pass re-downloaded the canonical
messages, found the local state consistent with L1, and never came back to those blocks.
…hes below the checkpointed tip

The leaf-count predicate is only exact while the removed index sits at or above what the published chain already
consumed. Below it every proposed descendant satisfies it, so the prune would drop blocks that consumed nothing
removed while the checkpointed parent that did consume them stays.
…cks are no longer local

P2P validates a checkpoint proposal and attests to it in two separate calls, so an archiver rollback in between can
prune the blocks a valid verdict was based on and the attestation would still be signed off the cached result.
…al message store

The previous version mocked the bucket lookup to a different rolling hash, which is what the neighbouring
mismatch test already covers. Rewinding a real message store and refilling the bucket with a different message
exercises the rollback, the bucket resolution and the re-read after the forced sync.
…what the prune actually covers

Adds inbox_rollback to the prune-type label values so the counter reports it from zero, records that a reorg
re-mining identical messages prunes blocks whose trees are unchanged, and trims the doc comments and log lines.
Waits for the withheld propose and checks the consuming block sits above the checkpointed tip before reorging,
identifies the prune by the blocks its event carries rather than by the tip having moved, asserts published state
was not unwound, and requires a checkpoint built after the reorg to reach L1.
…ccount

The inbox reorg e2e withheld the replacement message's L1 tx and handed it to
anvil_reorg to be mined in the block replacing the orphaned one, but both
messages were sent from the same account: the reorg rewinds past the first
message's tx, so the sender's nonce drops back below the withheld tx's, anvil
drops it from the replacement block, and the test times out waiting for its
receipt with the inbox left empty.

Send the replacement from a second account, whose nonce the reorg does not
touch. createL1Client handed out the same key on every call, so give it a
counter and let each client have its own account.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant