feat(fast-inbox): prune the proposed chain when an L1 reorg orphans consumed inbox messages - #25361
Draft
spalladino wants to merge 11 commits into
Draft
feat(fast-inbox): prune the proposed chain when an L1 reorg orphans consumed inbox messages#25361spalladino wants to merge 11 commits into
spalladino wants to merge 11 commits into
Conversation
…olls back consumed inbox messages
…archiver rewound the bucket
…sumed inbox messages
…n one transaction Splitting the two left the prune unrecoverable: a crash or a failure after the messages were dropped left the proposed chain built on messages the store no longer held, and the next sync pass re-downloaded the canonical messages, found the local state consistent with L1, and never came back to those blocks.
…hes below the checkpointed tip The leaf-count predicate is only exact while the removed index sits at or above what the published chain already consumed. Below it every proposed descendant satisfies it, so the prune would drop blocks that consumed nothing removed while the checkpointed parent that did consume them stays.
…ox rollback prune
…cks are no longer local P2P validates a checkpoint proposal and attests to it in two separate calls, so an archiver rollback in between can prune the blocks a valid verdict was based on and the attestation would still be signed off the cached result.
…al message store The previous version mocked the bucket lookup to a different rolling hash, which is what the neighbouring mismatch test already covers. Rewinding a real message store and refilling the bucket with a different message exercises the rollback, the bucket resolution and the re-read after the forced sync.
…what the prune actually covers Adds inbox_rollback to the prune-type label values so the counter reports it from zero, records that a reorg re-mining identical messages prunes blocks whose trees are unchanged, and trims the doc comments and log lines.
Waits for the withheld propose and checks the consuming block sits above the checkpointed tip before reorging, identifies the prune by the blocks its event carries rather than by the tip having moved, asserts published state was not unwound, and requires a checkpoint built after the reorg to reach L1.
…ccount The inbox reorg e2e withheld the replacement message's L1 tx and handed it to anvil_reorg to be mined in the block replacing the orphaned one, but both messages were sent from the same account: the reorg rewinds past the first message's tx, so the sender's nonce drops back below the withheld tx's, anvil drops it from the replacement block, and the test times out waiting for its receipt with the inbox left empty. Send the replacement from a second account, whose nonce the reorg does not touch. createL1Client handed out the same key on every call, so give it a counter and let each client have its own account.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the last gap in the node's recovery from an L1 reorg that orphans Inbox messages a block already consumed: the locally proposed chain built on those messages.
The problem
The archiver already rolls back orphaned messages and bucket snapshots (#25354), and the proposer already re-resolves the bucket it consumed before signing and before publishing (#25355). Neither touches the blocks this node already built. Under pipelining those blocks — and the checkpoint that covers them — live in the archiver as proposed state, so after the rollback the node still holds a pending chain whose L1-to-L2 tree contains leaves that no longer exist on L1. Recovery was indirect: the
proposereverts, the slot ends without a checkpoint, and the end-of-slot prune throws the blocks away one slot later. In the meantime the proposer of the next slot resolves the parent's bucket by cumulative message total, and after a re-sync a different bucket can carry the same total, so it can build a child on the wrong L1-to-L2 tree.The rule
A block's L1-to-L2 tree leaf count is the cumulative count of messages it consumed through — that identity is what the whole streaming design rests on. So when the rollback removes messages from index
fon, a proposed block whose leaf count is> fconsumed at least one removed message; the first such block starts the invalid suffix, and everything from it on goes.fis the index the rollback rewound to, not the first leaf whose value actually changed, so a reorg that re-mines the same messages in a different L1 block also prunes blocks whose trees are in fact unchanged. That over-pruning is deliberate rather than incidental: the bucket those blocks reference was re-timed and re-numbered by the reorg, so L1 would reject a checkpoint carrying it anyway.The message rewind and the block prune commit in a single store transaction. Splitting them would make the prune unrecoverable — a crash in between leaves a proposed chain built on messages the store no longer holds, and the next pass re-downloads the canonical messages, finds the local state consistent with L1, and never comes back to those blocks.
The prune runs inside the message step of the sync pass, so it fires before the checkpoint step and every consumer — world state through the tips, the sequencer and validators through their pull checks — sees a consistent local view after a single poll. It is idempotent: a second rollback at the same index finds nothing left to prune.
Two boundaries the predicate gets right by construction: a block whose leaf count equals
fconsumed only indices below it and is kept, and a block that consumed no messages of its own is pruned only when its parent is.Published checkpoints are never touched by this path. If the message store disagrees with a checkpoint L1 accepted, one of the two views of L1 is mid-reorg and this is not necessarily the right one — L1 validates the rolling hash in its own chain, so the archive comparison in the checkpoint step stays the authority on published state. Pruning it on a message signal alone would let a flaky RPC unwind real chain. For the same reason nothing is pruned when
ffalls below the checkpointed tip's leaf count: there every proposed descendant satisfies the predicate, including ones that consumed nothing removed, and acting on it would drop them while the published parent that did consume those messages stays.The pruned blocks go out on the existing
L2PruneUncheckpointedevent and are counted under a newinbox_rollbackprune-type label; world state follows the tips moving backwards, as it does for every other prune.Validators
The bucket check resolves the reference locally and compares rolling hashes, so once a validator's own archiver has rewound, a proposal on the orphaned bucket is rejected with
bucket_hash_mismatch— including after the single forced re-sync the handler does on a mismatch. Added a test that pins exactly that, driven off a real message store so the rollback, the bucket resolution and the post-sync re-read are the production ones.One validator-side fix: checkpoint proposal validation caches its verdict by signed-payload hash, and p2p makes two calls for one proposal (the all-nodes validation, then the attestation). A rollback in between prunes the blocks a valid verdict was based on, and the attestation would still be signed off the cached result. A valid cached verdict is now only reused while the checkpoint's last block is still local.
Known gap
A block whose world-state fork was taken before the rollback can still be pushed into the archiver after the prune scanned it. The sequencer forks once per slot and pushes each block it builds as it goes; the validator has the analogous window after re-execution. Such a block lands on the parent that survived the prune, its messages are canonical again by then, and no later prune fires. Closing it exactly needs the builder to carry the message-store state it built against so the insert can reject a stale one; a plain "a proposed block may not consume more messages than the store holds" assertion is correct in production but is violated by the random block fixtures the archiver test suites are built on, so it is not a change that belongs here. Marked with a
TODOat the insert site. In practice the block still cannot be published — the proposer re-resolves the bucket by rolling hash before publishing and abandons the slot when it no longer resolves — so the exposure is a stale proposed tip that the end-of-slot prune clears, which is where this stood before this PR.Tests
archiver-sync: a rollback prunes from the first proposed block past the removed index and keeps the ones at or below it (boundary and message-free-successor cases included), evicts the proposed checkpoint covering them, and moves the tips world state follows; a reorg that refills the same indices with different leaves still prunes, since the block consumed the old leaves; a prune that fails leaves the messages in place, so the next pass retries both halves together; checkpointed blocks survive a rollback of the messages they consumed; and a rollback reaching below the checkpointed tip prunes nothing.proposal_handler: the validator rejection described above, and the cached-verdict re-validation.single-node/cross-chain/streaming_inbox_reorg) that withholds the checkpoint'sproposetx so the consuming blocks are still proposed, checks the consuming block sits above the checkpointed tip before reorging, reorgs the message out and mines a replacement in its place, and asserts the prune event covers the consuming block within a fraction of a slot, published state was not unwound, the orphaned message is not consumable, the replacement is, and a checkpoint built after the reorg reaches L1. This e2e has not been run — it compiles and is wired into CI by the category glob, but it was written without a local run.No L1 change, no configuration change.
Fixes A-1389
Stacked on #25355