RFC 0013 addendum: application-consistent recovery for scale-to-zero

[giodl73-repo]

Summary

Defines the application-consistency contract a host needs to scale OpenClaw
compute to zero and later admit work on a replacement. It adds three optional,
host-neutral lifecycle specifications to the completed RFC 0013 SQLite
snapshot contract:

• recovery-point component composition;
• portable final handoff; and
• restored admission.

Together they define the application-consistency proof a host needs before it
can safely replace a Gateway generation or remove idle compute. They do not
define a new snapshot implementation, pause API, storage provider, or wake
system.

This PR is stacked on RFC PR #20.
RFC 0013 and
openclaw/openclaw#105718
remain authoritative for one-database SQLite snapshot creation, verification,
local publication, and fresh-target restore.

Why This Is Needed

Per-user and event-driven hosts can scale resident compute to zero only if they
can prove all of the following:

1. the retiring Gateway generation closed the complete owner-selected state;
2. the host durably accepted the exact required byte set;
3. source-compute retirement was authorized for that exact generation; and
4. replacement admission remained closed until the same recovery point and
   required owner state were restored.

A verified SQLite snapshot proves none of those aggregate lifecycle facts by
itself. Without a shared contract, hosts must either keep idle Gateways warm or
maintain private shutdown, copying, restore-ordering, and readiness-inference
paths.

The motivating managed-host deployment already has durable ingress and cron
wake machinery. Its production issue cluster includes accepted work outliving
the compute that should process it, cold-start head-of-line blocking, cron wake
racing idle retirement, scheduler continuity failing after replacement, and
cold runtimes appearing ready before the required owner state is restored.
OpenClaw's broader state-ownership risk is also tracked in
openclaw/openclaw#101290.
The complete user/operator case, including missed recurring work, competitor
scale-to-zero primitives, and Microsoft Scout's host-side composition, is
tracked in
openclaw/openclaw#114145.

Industry Baseline And OpenClaw's Missing Layer

Infrastructure and sandbox platforms already expose the compute primitives:

Platform	Primitive
Fly Machines	Traffic-driven stop-to-zero and autostart
E2B	Persistent pause and automatic resume
Daytona	Persistent stop/start, archive, and VM pause/resume
Azure Container Apps	Event-driven scale-to-zero
Cloudflare Durable Objects	Hibernation with wakeable WebSocket delivery

Those systems cannot infer OpenClaw's complete owner inventory, SQLite
invariants, scheduler state, or restored-readiness boundary. This addendum
defines that application-consistent layer. Retained ingress, wake registration,
compute placement, and external durability remain host responsibilities.

What Changed From The Earlier Continuity Direction

The sidecars compose contracts already shipped on OpenClaw main:

• openclaw backup sqlite from
  #105718; and
• gateway.suspend.prepare|status|resume from
  #103618, including
  Vincent's validation repair in
  #103925.

They replace the earlier broad continuity direction rather than adding another
SQLite capture path, Gateway pause API, provider framework, Lobster transport,
CAPE configuration surface, or Elastic wake mechanism.

The suspension contract remains a cooperative tracked-work fence, not full
process quiescence. The host still owns external ingress fencing, third-party
Channel transports, unregistered work, process supervision, encryption,
retention, placement, and wake.

Closed Boundaries

• The runtime state owner supplies a canonical required-component inventory;
  missing and extra components fail closed, and the inventory is covered by
  the recovery-point identity.
• Ordinary RFC 0013 snapshots remain host-protected. They cannot be called
  credential-free portable without exact owner-authored portability receipts.
• Obligations use a closed kind/owner/treatment vocabulary and contain no
  secret values.
• Host acceptance binds canonical aggregate-manifest bytes plus every exact
  owner-manifest and artifact byte sequence. Packaging and paths remain
  host-owned.
• Restored admission binds the same acceptance-set identity through hold,
  restored-start, and ready evidence. When RFC 0018 canonical readiness is
  available and enabled, it projects that same evidence without adding a
  second evaluator.
• Unknown outcomes hold or quarantine. Quarantine has no automatic exit.

The lifecycle facts are deliberately separate:

GatewaySuspensionReady
  -> SourceWritersClosed
  -> RecoveryPointCaptured
  -> RecoveryPointAccepted
  -> SourceComputeRetirementAuthorized
  -> RecoveryPointRestored
  -> AdmissionReady

There is no global /synced state. Each fact has a different owner, subject,
lifetime, and failure boundary. The diagram is the success path, not an
unconditional linear workflow: retained work may race retirement, atomically
revoking it while source compute exists or joining the destination wake after
retirement.

Observable Outcome Of The Complete Composition

• Idle resident compute may reach zero without treating raw live files as a
  recovery point.
• Accepted ingress remains retained until restored admission succeeds.
• Autonomous scheduled work can recover from absent compute without an
  unrelated user message.
• Source compute is never retired before exact durable acceptance.
• Replacement readiness identifies the accepted recovery point it restored.

These are system-level acceptance criteria. The three current OpenClaw PRs
prove only the owner-side recovery-point, final-capture, and restored-admission
slices; they do not claim to complete host acceptance, retained ingress, wake,
or source-compute retirement.

One Contract, Two Directions

The lifecycle can be presented as one bidirectional host/runtime contract
without putting host lifecycle authority in OpenClaw. OpenClaw supplies the
running-source suspension methods, offline final-capture and accepted-restore
operations, and the restored-readiness fact. The host supplies durable
acceptance, compute generations, retained wake, placement, and source-compute
retirement.

Quiesced
  -> RecoveryPointCaptured
  -> RecoveryPointAccepted
  -> generation-scoped SourceComputeRetirementAuthorized
  -> retained wake cause
  -> RecoveryPointRestored
  -> AdmissionReady
  -> owner delivery and acknowledgement

This is a protocol boundary, not a generic base class or new callback
framework. A host may compose it behind prepareHibernate,
ensureRuntimeReady, and inspectLifecycle, but those remain host operations.
The existing gateway.suspend.*, scheduler hooks, recovery operations, and
admission fence are reused.

The RFC now recommends one concrete missing seam:

gateway.restore.status({ restoreOperationId })
  -> { status: "not-restored" }
   | { status: "held", reason, retryAfterMs, ...restoreIdentities }
   | { status: "ready", ...restoreIdentities, schedulerIdentity,
       ownerReadinessIdentity, readinessIdentity }

This follows Peter's gateway.suspend.status pattern: closed Gateway Protocol
schemas, operator.read, stable operation identity, no mutation, exact replay,
and typed conflict behavior. It projects PR 112896's existing durable ready
record rather than adding another journal or state machine. Offline restore
and quarantine do not become invented live phases. The method remains callable
only through an authenticated pre-admission control path while ordinary work
is held. That path is a tracked, bounded, non-enrolling host probe: restart
still wins, and nodes, pairing, remote clients, presence-bearing sessions, and
ordinary RPCs remain fenced. There is no gateway.restore.admit; OpenClaw
computes readiness and opens its own admission.

When RFC 0018 is available and enabled, the same record also publishes a
required RecoveryPointRestored condition through /readyz, openclaw ready,
Status, and Gateway readiness RPC. Stable subject refs identify the destination
Gateway, restore operation, accepted recovery point, runtime generation,
scheduler reconciliation, and required state-owner roles; current IDs and
generations stay in RFC 0018's identity package. The generic readiness
projection and operation-fenced status method must report the same readiness
generation. Neither may infer restore success from process health or
database-open success. gateway.restore.status remains independently useful
when canonical readiness is disabled or has not landed.

Planned handoff may create a new recovery point only after Gateway suspension,
host-proven source-writer closure, final capture, and durable acceptance.
Forced source loss creates no clean-retirement fact and recovers from the last
previously accepted point under its declared RPO.

Implementation Evidence

• openclaw/openclaw#112385
  composes verified RFC 0013 snapshots into one deterministic host-protected
  recovery point and exact acceptance-byte inventory, using one canonical
  generation-bound state-owner inventory receipt across the stack.
• openclaw/openclaw#112865
  is the final-capture PR stacked on #112385. It binds
  host-supplied closure evidence, creates final owner snapshots, and durably
  replays one committed recovery point.
• openclaw/openclaw#112896
  is the restored-admission PR stacked on #112865. It restores an exact
  accepted recovery point to fresh paths and reuses Gateway work admission to
  remain closed until scheduler reconciliation and required-owner readiness
  produce one durable ready record. It now also implements the proposed
  read-only gateway.restore.status projection and its tracked local probe
  handshake as representative proof. The
  public Gateway Protocol addition remains an explicit owner-review decision
  before merge or ship.
• Fork-only Lobster drafts
  #39,
  #41,
  #42,
  #43, and
  #45,
  #46,
  #47, and
  #51 (Microsoft access
  required) demonstrate permanent cron projection, cancellation arbitration,
  Peter's shipped hook/proxy-pipe composition, and a default-off private
  Blob/per-user ECS/already-due sleep-veto boundary, followed by one permanent
  payload-free retained-wake authority that coalesces concurrent owner causes
  into an exact destination generation, production-backed metadata-only
  Teams/cron owner validation, and exact-readiness-gated per-cause delivery
  with durable crash replay.

The retained-wake drafts deliberately do not activate a production route or
cohort. Lobster still needs an owner-approved accepted-final-handoff lookup and
canonical mapping from opaque logical runtime identity to the Teams/cron
semantic owners before those call sites can be wired safely.

These PRs and fork drafts are non-normative feasibility and owner-review
evidence. None moves external ingress fencing, clean shutdown, host acceptance,
publication, wake, compute placement, or source-compute retirement into
OpenClaw.

If Vincent, Peter, Josh Lehman, or other maintainers prefer a smaller or
different implementation that meets the RFC and issue #114145 outcomes, please
use it. The fork stack is evidence for review, not a requirement to preserve
this exact decomposition.

Review Request

Please review the ownership split and public status method before the
implementation stack is promoted:

1. RFC 0013 remains the completed SQLite owner primitive.
2. Current-main gateway.suspend.* remains the cooperative pause primitive.
3. The state owner, not aggregate composition, supplies the complete selected
   agent inventory from the activation-pinned runtime and binds its revision to
   the source generation.
4. Ordinary snapshots remain host-protected; portability requires owner proof.
5. Acceptance binds a closed logical byte set without standardizing a storage
   container.
6. The host owns ingress fencing, suspension orchestration, clean shutdown,
   wake, and compute lifecycle; final capture only binds supplied closure
   evidence.
7. OpenClaw core owns recovery-journal schema, migration, corruption, repair,
   retention, and cleanup semantics; unknown journal state never falls back to
   ordinary startup.
8. When available and enabled, RFC 0018 owns the canonical current-readiness
   projection, while gateway.restore.status remains the independent exact
   operation-fenced observation seam.

[giodl73-repo]

Hey @vincentkoc — this is the small follow-on referenced from #20.

The review question is only the ownership split:

1. RFC 0013 and backup sqlite remain authoritative for each SQLite artifact.
2. gateway.suspend.* remains the cooperative tracked-work fence.
3. These sidecars add only aggregate recovery-point identity, durable handoff acceptance, and restored admission/replay above those shipped contracts.

We are holding implementation until that boundary is accepted. The earlier RFC 0021 / twelve-PR design is now retained only as evidence, not proposed as the path forward.

[giodl73-repo]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/30010475441
• Updated: 2026-07-23T13:16:35.194Z

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 28, 2026, 10:41 AM ET / 14:41 UTC.

ClawSweeper review

What this changes

The branch adds three RFC 0013 follow-on specifications for composing recovery points, accepting a final handoff, and keeping Gateway work admission closed until an accepted recovery point has been restored and verified.

Merge readiness

⚠️ Ready for maintainer review - 4 items remain

Keep this PR open. It is a clean, documentation-only RFC addendum stacked on the still-open RFC 0013 proposal, and it introduces a proposed public Gateway restore-status contract that needs explicit protocol and product approval rather than cleanup closure.

Priority: P3
Reviewed head: 2f290e9b3f09708bb309d6c4c9bd2e39dc76b663
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The documentation is substantial and internally scoped, but merger confidence is limited by an unresolved public Gateway protocol and product-direction decision.
Proof confidence	🌊 off-meta tidepool	Not applicable: All four changed files are RFC documentation, so after-fix runtime proof is not required for this patch; linked implementation PRs remain supporting context rather than proof for the RFC text.
Patch quality	🦐 gold shrimp (3/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: All four changed files are RFC documentation, so after-fix runtime proof is not required for this patch; linked implementation PRs remain supporting context rather than proof for the RFC text.
Evidence reviewed	5 items	Stacked RFC dependency: This PR targets the open RFC 0013 branch at 864d7dee6ba6b9cf9e2f902f174408e5fbc570db, not main; the parent RFC remains open, so neither the baseline RFC nor this addendum is already implemented on the default branch. Concrete documentation scope: The head changes four RFC Markdown files: one optional-composition extension to RFC 0013 and three draft sidecar specifications for component composition, portable handoff, and restored admission. Unresolved protocol decision: The restored-admission specification proposes gateway.restore.status but explicitly describes its public Gateway Protocol addition as pending owner review; the contributor’s discussion likewise asks for an ownership-boundary decision before implementation proceeds.
Findings	None	None.
Security	None	None.

How this fits together

RFC 0013 defines SQLite-safe snapshot artifacts, while these sidecars describe how a host and OpenClaw can compose those artifacts during planned handoff and scale-to-zero recovery. Hosts provide durable acceptance, retained ingress, and compute placement; Gateway recovery and readiness evidence determine when normal work may resume.

flowchart LR
  A[Gateway tracked work] --> B[Source writers closed]
  B --> C[Recovery point captured]
  C --> D[Host accepts exact artifacts]
  D --> E[Fresh Gateway restore]
  E --> F[Owner and scheduler readiness]
  F --> G[Work admission opens]

Loading

Decision needed

Question	Recommendation
Should the RFC adopt an additive public gateway.restore.status contract now, or keep restored-admission evidence host-private until the Gateway protocol owner approves a stable public API?	Defer the public API commitment: Keep the recovery-point, handoff, and admission responsibilities in the RFC, but state that any Gateway status method remains non-normative until a protocol-owner proposal is separately accepted.

Why: The patch is documentation-only and has no definite implementation defect, but it proposes externally visible Gateway behavior and a long-lived compatibility boundary; that contract choice cannot be settled by mechanical review.

Before merge

• Add real behavior proof - Not applicable: All four changed files are RFC documentation, so after-fix runtime proof is not required for this patch; linked implementation PRs remain supporting context rather than proof for the RFC text.
• Resolve merge risk (P1) - Merging would publish a new Gateway Protocol-shaped gateway.restore.status direction before the protocol owner has accepted its compatibility and pre-admission control boundary.
• Resolve merge risk (P1) - The addendum depends on the still-open RFC 0013 branch, so its final wording and merge order must remain aligned with the parent RFC rather than being reviewed as an independent completed contract.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
RFC documentation surface	4 files affected; 1,350 added, 1 removed	The patch is a substantial contract addition rather than a small correction, so approval should focus on its durable API and responsibility boundaries.
Runtime code changes	0 files	All changed paths are RFC Markdown, making runtime proof inapplicable for this documentation patch while leaving product-direction review necessary.

Merge-risk options

Maintainer options:

1. Defer the protocol commitment (recommended)
   Keep the recovery lifecycle documentation but make the proposed restore-status endpoint explicitly non-normative until the Gateway protocol owner selects a compatibility contract.
2. Approve the additive API boundary
   A protocol owner may accept the read-only status projection now if its authorization, held-state, and upgrade semantics are intentionally part of the public Gateway contract.
3. Pause the addendum with the parent RFC
   Do not merge this stacked proposal if the parent SQLite RFC or the ownership split is not ready for a scale-to-zero follow-on commitment.

Technical review

Best possible solution:

Accept or narrow the host/runtime responsibility split with the Gateway protocol owner, preserving the SQLite snapshot contract as the parent boundary and treating any public restore-status API as an explicitly approved additive follow-on.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is an RFC documentation proposal, not a report of a current runtime failure. Its linked implementation PRs are supporting evidence, but the proposed contract itself requires design review rather than bug reproduction.

Is this the best way to solve the issue?

Unclear: the sidecar split is a coherent way to keep host lifecycle authority outside the SQLite snapshot contract, but the public gateway.restore.status seam is still an explicit protocol-owner choice and should not be treated as settled by this RFC alone.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 593e6e46632b.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P3: This is a forward-looking RFC and documentation proposal with no current runtime regression or user-facing outage.
• merge-risk: 🚨 compatibility: The proposed Gateway restore-status method could establish a public protocol and authorization compatibility commitment if merged as normative RFC guidance.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: All four changed files are RFC documentation, so after-fix runtime proof is not required for this patch; linked implementation PRs remain supporting context rather than proof for the RFC text.

Evidence

What I checked:

• Stacked RFC dependency: This PR targets the open RFC 0013 branch at 864d7dee6ba6b9cf9e2f902f174408e5fbc570db, not main; the parent RFC remains open, so neither the baseline RFC nor this addendum is already implemented on the default branch. (rfcs/0013-cloud-serializable-sqlite-state.md:1, 2f290e9b3f09)
• Concrete documentation scope: The head changes four RFC Markdown files: one optional-composition extension to RFC 0013 and three draft sidecar specifications for component composition, portable handoff, and restored admission. (rfcs/0013-cloud-serializable-sqlite-state.md:173, 2f290e9b3f09)
• Unresolved protocol decision: The restored-admission specification proposes gateway.restore.status but explicitly describes its public Gateway Protocol addition as pending owner review; the contributor’s discussion likewise asks for an ownership-boundary decision before implementation proceeds. (rfcs/0013/restored-admission-v1-spec.md:1, 2f290e9b3f09)
• Prior review continuity: The previous completed review cycle found no line-level defect. The current head remains documentation-only, so real runtime proof is not applicable to this RFC patch itself. (rfcs/0013/restored-admission-v1-spec.md:1, 2f290e9b3f09)
• Repository-policy inspection limitation: The read-only execution sandbox failed before any command could run (bwrap: loopback: Failed RTM_NEWADDR), so the checkout’s AGENTS.md and local git history could not be read in this review; this verdict therefore relies on the supplied complete PR metadata, diff, discussion, and history context.

Likely related people:

• giodl73-repo: The contributor authored both the open RFC 0013 parent proposal and this addendum, and the addendum’s stated responsibility split is the direct continuation of that RFC work. (role: author of the parent RFC and follow-on specifications; confidence: high; commits: 17fcd5d894d7, 2f290e9b3f09; files: rfcs/0013-cloud-serializable-sqlite-state.md, rfcs/0013/recovery-point-components-v1-spec.md, rfcs/0013/portable-handoff-v1-spec.md)
• vincentkoc: The contributor explicitly requested review of the ownership split from vincentkoc, and the proposal builds on the SQLite snapshot implementation attributed to Vincent in the parent RFC context. (role: adjacent snapshot-contract reviewer; confidence: medium; commits: 2f25e9cba384; files: rfcs/0013-cloud-serializable-sqlite-state.md)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Obtain an explicit Gateway protocol-owner decision on whether gateway.restore.status is part of the intended public contract.
• Keep the addendum’s final wording aligned with the still-open RFC 0013 parent proposal.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (21 earlier review cycles; latest 8 shown)

• reviewed 2026-07-27T07:49:17.515Z sha aaa077a :: needs real behavior proof before merge. :: none
• reviewed 2026-07-27T14:19:43.241Z sha cdbee3b :: needs real behavior proof before merge. :: none
• reviewed 2026-07-27T17:13:58.889Z sha 6205f9d :: needs real behavior proof before merge. :: none
• reviewed 2026-07-27T18:01:15.155Z sha 6205f9d :: needs real behavior proof before merge. :: none
• reviewed 2026-07-27T21:13:45.005Z sha bb46179 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-27T21:24:22.645Z sha 1403542 :: needs real behavior proof before merge. :: none
• reviewed 2026-07-27T22:16:55.758Z sha 88752bd :: needs real behavior proof before merge. :: none
• reviewed 2026-07-28T04:09:00.944Z sha 37d390b :: needs real behavior proof before merge. :: none

[giodl73-repo]

@clawsweeper re-review

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.