feat(fast-inbox): wait for the local archiver before rejecting a block proposal for an unsynced inbox bucket

[spalladino]

Stacked on #25323.

The race

A validator receiving a block proposal first runs the streaming-Inbox metadata checks, which look the proposal's bucket up in the validator's own archiver. If the archiver has not yet synced the L1 block that opened that bucket, the lookup returns undefined and the proposal was rejected on the spot with bucket_unknown.

That is a pure race, not a divergence. The proposer only consumes buckets at least one Ethereum slot old, so the bucket is on L1; the validator's archiver polls L1 on an interval, and a proposal arriving inside that window lost this validator's attestation. With buckets opening in nearly every L1 block under load this is a steady drip of lost attestations, not an edge case.

What waits and what does not

• bucket_unknown → bounded wait. Force an archiver sync, re-run the whole metadata check, repeat every 0.5s until it resolves or the slot's attestation deadline (target_slot_start + S − 2E) passes — the same bound the handler's other sync waits use. On timeout the rejection keeps the bucket_unknown reason (no new reason string to teach the slashing/invalid classification maps) plus a warn carrying reason: 'bucket_sync_timeout' and waitedMs.
• bucket_hash_mismatch → one forced sync and one re-check. A known bucket seq with a different rolling hash means the two nodes saw different L1 blocks at that height, but it does not say which side is stale: after an L1 reorg the proposer may already be on the canonical replacement while this validator still holds the orphaned bucket. One forced sync performs the rollback and re-sync if we were the stale side. If the hashes still differ, our view is as good as L1's and the proposal is on the wrong fork — reject, with a bucket_hash_mismatch_after_sync warn. No loop: a persistent mismatch will not resolve by waiting.
• Everything else rejects immediately (bucket_too_new, bucket_moves_backwards, the caps, parent_bucket_unresolved, a proposal with no bucketRef, and any arrival after the deadline has already passed — in that last case the archiver is not poked at all).
• The wait re-runs checkStreamingBlockMetadata rather than just the bucket lookup, so it also covers the case where "the block before the checkpoint's first block has not synced" maps to bucket_unknown — the same kind of local lag — and guarantees the accepted result was computed against the synced view.

Shared wait helper

The handler already had three near-identical retryUntil(syncImmediate + lookup, { deadline, dateProvider }, 0.5) blocks. Rather than adding a fourth, they now share a private awaitLocalSync(slot, what, resolve) that owns the deadline computation, the past-deadline short-circuit, the forced sync per attempt, and the TimeoutError → undefined conversion. Callers keep their own log level and their own timeout fallback value.

Migrated (behavior preserved exactly, proposal_handler.test.ts green with no test changes before any of the bucket work landed — 45/45):

• getParentBlock (parent block not yet synced). Keeps its own past-deadline check so the "timed out" debug log still fires only for real timeouts, not for an arrival with no budget left.
• resolveExistingBlockAtNumber (stale fork at this block number during a reorg). Already had the past-deadline guard ahead of its warn, so the mapping is exact.

Not migrated:

• The checkpoint proposal's last-block sync wait in validateCheckpointProposal. It has no past-deadline guard, and retryUntil runs fn once before checking the timeout — so today, when a proposal arrives past the deadline, it still performs one sync + lookup and accepts an already-synced block (there is a test pinning exactly that). The shared helper short-circuits to undefined in that case, which the new bucket wait needs, so migrating this site would flip that case to last_block_not_found. Left as is.

DoS bound

Confirmed the p2p retention bound rather than assuming it. The attestation pool caps distinct payload hashes per position at MAX_BLOCK_PROPOSALS_PER_POSITION = 2 per (slot, indexWithinCheckpoint), and gossip validation rejects indexWithinCheckpoint >= MAX_ATTESTABLE_BLOCKS_PER_CHECKPOINT (= MAX_BLOCKS_PER_CHECKPOINT = 72) at ingress, after checking that the proposal is signed by the slot's expected proposer. So the worst case is ≤ 144 concurrently waiting handlers per slot, all from the elected proposer for that slot, each polling a RunningPromise.trigger that coalesces concurrent requests into a single sync run (one L1 head query when nothing changed). That is small, so the cheap indexWithinCheckpoint validation was not hoisted ahead of the wait and the handler's order is unchanged. The cost of a bucket that never appears is the proposer's own slot, which it could waste anyway by not proposing.

Tests

Six new unit cases in proposal_handler.test.ts (bucket sync wait), all asserting on the result rather than on call counts, except where "did we poke the archiver at all" is the behavior. retryUntil converts the deadline to a duration once and then runs on a real timer, so the deadline cases hold the fake clock 2s short of the slot-1 attestation deadline (40s) and assert "within one interval after", never "exactly at"; the whole block runs in ~2.6s.

Red/green on case 1 and the mismatch cases, before the handler change:

✕ attests once the referenced bucket shows up on a later archiver sync
    expect(result.isValid).toBe(true) → Received: false   (rejected with bucket_unknown)
✕ rejects with bucket_unknown when the bucket never syncs, no earlier than the deadline
    expect(elapsedMs).toBeGreaterThanOrEqual(2000) → Received: 3   (no wait at all)
✓ rejects immediately without syncing when the attestation deadline has already passed
✓ rejects immediately without syncing when the proposal carries no bucket reference
✕ rejects a hash mismatch that survives one forced sync, without looping
✕ attests when the forced sync replaces our stale bucket with the proposed one
Tests: 4 failed, 2 passed

After:

Tests: 51 passed, 51 total   (proposal_handler.test.ts: 45 pre-existing + 6 new)

Full gate, all from yarn-project:

• yarn build — clean
• yarn format validator-client, yarn lint validator-client — clean
• yarn workspace @aztec/validator-client test — 274 passed, 3 skipped, 277 total (10 suites)

The existing streaming_inbox_checks.test.ts case "rejects promptly when the referenced bucket is unknown (no waiting)" is unchanged: the pure check is still immediate, the wait lives in the handler. Stale comments in streaming_inbox_checks.ts and checkStreamingBlockMetadata that said there was no bounded wait yet are updated.

No e2e was added: the behavior is a handler-local retry over an already-tested archiver API, and covering it end to end would need a test-only archiver pause hook (Archiver.stop() cannot stall RunningPromise.trigger()) plus a committee-member selection dance for minutes of wall clock per case.