[Luomu (cloudonly)]

Revert deletes committed local checkpoints while their async uploads are still PENDING. It can restore an older snapshot or cold-start instead. SUSPENDED admission also changes with snapshot locality.

Fixes #2983.

• Preserve committed local snapshots and pending uploads during workload teardown, return the retained state from atelet, and reschedule detach.
• Reconcile the retained state before cleanup, including failed Suspend cleanup and detach completed while the actor was crashed. Reject missing tracker state unless durability is proven, and clear locality when the assigned node is gone.
• Retain the latest local checkpoint and durable fallback. Reject Revert on SUSPENDED actors regardless of locality, and update API and CLI documentation.
• Keep regressions for incomplete checkpoint cleanup and recovery of the previous snapshot. The moveLocalCheckpoint implementation now comes from main.

Rebased onto the latest main.

Validation on the rebased branch:

• Related Revert, Resume, controlapi, functionaltest, atelet, and snapshottracker regressions passed with -race -count=3 -shuffle=on: 1,551 passing test and subtest entries, without skips. Tests ran as a regular Linux user. Functional tests used PostgreSQL, envtest, and a fake atelet.
• Path regressions in nodepath and ateletpath passed with -race -count=3 -shuffle=on: 42 passing test and subtest entries.
• go build ./cmd/atelet, hack/verify/gofmt.sh, and git diff --check passed.
• Real gVisor Revert/Resume, CRASHED recovery, and Delete E2E passed three shuffled runs: 15 JUnit cases, with no failures or skips. Images were built from the rebased head with the external Linux 5.14 OverlayFS compatibility overlay.

Before this rebase, cross-filesystem snapshot GC and Terminate retention and cleanup tests passed inside an isolated kind node with -race -count=3 -shuffle=on: 48 passing entries, without skips.

An earlier broader atelet and snapshottracker run found a data race in the unchanged fakeJWTMinter test fixture. The same isolated test reproduced it on main 2864a31f in a 500-run comparison. The related regression run above passed. The earlier broader run is not a clean pass.

Earlier validation on 1ffedeba: full package race tests passed with 11,240 test and subtest entries, using -vet=off and an external Linux 5.14 OverlayFS compatibility overlay. Filecache ran separately without DAC overrides. Fifteen skips matched the parent; nine permission cases passed separately as a regular user. go build ./cmd/... and all hack/verify/ scripts passed. These full checks have not been repeated on the rebased head.

microVM E2E was not run because the validation host has no KVM. The kernel compatibility overlay stays outside this PR.

AI was used for conflict resolution, review, the cleanup fix, and test validation.

The rebase also handles failed cross-node restores: Revert clears locality only for confirmed deleted snapshot nodes, preserves live-node copies, and retains PENDING metadata until normal cleanup. Snapshot-node lookup errors stop cleanup; Actor version conflicts remain protected. Six regression cases cover these boundaries.

Latest rebase (2026-10-23): rebased onto main 4ebd668d at 8678da7e. Focused race regressions passed three shuffled runs; affected binaries and test packages compiled. Earlier validation entries describe prior revisions. Cluster E2E was not rerun. Codex assisted with this rebase and validation.

[Haven Xia (HavenXia)]

I think #3079 is doing the same fix for #2983 but in a different way. Zoe Zhao (@zoez7) could you take a look?

[Chenyi Wang (chw120)]

Heads-up for the rebase: #2797, merged after this PR's base, added another Terminate caller that this PR doesn't update. When a worker's pod goes away, DeleteWorker reclaims the actor's node state with its own TerminateRequest (cmd/ateapi/internal/controlapi/workflow_worker_delete.go:348-356), which will still send preserve_snapshots: false. This PR doesn't touch that file, so the rebase won't flag it.

That breaks the case this PR's api-guide change covers. Say an actor resumes from a local snapshot whose upload is still pending, and then its worker pod goes away. The reclaim deletes that snapshot before the user can revert, and RevertActor then fails on every retry with FailedPrecondition: atelet on node N cannot account for local snapshot generation G, leaving the actor stuck in REVERTING. On main today the same sequence silently reverts to the previous snapshot. That's the loss #2983 reports, only triggered by DeleteWorker instead of by RevertActor itself.

Setting PreserveSnapshots: true in that request looks like the smallest fix. I found this by reading the code and haven't run it on a cluster.

[Luomu (cloudonly)]

Thanks, confirmed. reclaimActorStateOnNode sent a full Terminate, so atelet forgot the local snapshot while the actor kept its locality. applyRevertedSnapshotState then rejected the revert on every retry.

Pushed 40dc5fc, without a rebase. It sets PreserveSnapshots: true in that request and asserts it in TestDeleteWorkerWorkflow_ReclaimsActorNodeState.

One cost: with preserve, atelet empties the actor directory but keeps the snapshot directory until the detach delay passes, so this frees less disk right away than #2797 did. I checked the request with a store-free test and read the code. The Postgres-backed test didn't run on my machine, so CI is its first run.

[Luomu (cloudonly)]

Thanks for the pointer. The two PRs overlap but fix different cases. This one keeps a committed local snapshot and its pending upload when the actor is terminated for a revert. #3079 records an upload that atelet finished but ateapi had not yet recorded. Both change Terminate and the revert flow, so they will conflict.

Zoe Zhao (@zoez7), since #3079 is back in draft, could we agree on which lands first?