[nickdujay-pplx]

Summary

PR #583 stops a rolled-back restore from going live when Kubernetes retries it late. Unit tests cover each refusal. No test runs the real sequence on a cluster, because nothing in a running cluster makes a restore fail on demand.

This PR adds that end-to-end test. It is a Kind smoke scenario. It fails one restore on purpose, lets kubelet retry it late, and checks that the sandbox keeps its original snapshot.

Self-assessed severity: Medium. This PR changes no production behaviour. It adds a regression guard for a High-severity bug (see #583). Release builds do not compile the new hook.

Stack

This PR is the second in a gh stack:

1. sbx-mgr: skip teardown snapshot after a confirmed activation rollback #583 sbx-mgr: skip teardown snapshot after a confirmed activation rollback: the fix.
2. This PR: the end-to-end guard for that fix. Its base is the sbx-mgr: skip teardown snapshot after a confirmed activation rollback #583 branch.

The scenario needs the fix to pass. Review #583 first. This PR can merge only after #583, or together with it.

Terms

• Restore prepare: the step where sandbox-manager asks node-manager to set up the sandbox before it restores the snapshot.
• Activation attempt: one try at starting an execution generation. Kubelet retries the same attempt until the pod is deleted.
• Rollback: sandbox-manager undoes a failed restore. The sandbox is suspended again on its original snapshot.
• Test hook: an environment variable that only test-hooks builds read. Kind builds enable the feature; release builds do not.

Problem

The bug in #583 needs four things in order:

1. A restore fails after its activation claim.
2. Sandbox-manager rolls the attempt back.
3. Kubelet retries the same attempt late.
4. A teardown runs, and then a new connect.

Steps 1 and 3 do not happen on demand. A real prepare failure, such as a disk-admission refusal, needs a full disk. Filling the disk, or restarting a service, also moves placement or races the restore. So the Kind suite had no way to run this sequence, and a regression would pass CI.

The user's files survive even on the broken build. So a test that only reads the files back cannot catch the bug. The test has to check the snapshot rows and the restored artifact.

What changed

Sandbox-manager: a one-shot restore-prepare fault

New file: crates/libs/sbx/mgr/server/src/service/restore_prepare_fault.rs.

• With test-hooks, the hook reads SPACE_SBXMGR_TEST_RESTORE_PREPARE_FAULT_DIR.
• A file in that directory named exactly after a sandbox id arms one fault for that sandbox.
• The next restore prepare of that sandbox removes the file and fails with RESOURCE_EXHAUSTED. That is the code a real disk-admission refusal carries. The rollback path is unchanged.
• Removing the file is what fires the fault. So each file fires at most once, and two racing restores cannot both fail.
• An id that is not one plain path component never fires.
• Without test-hooks, the hook is an #[inline(always)] function that returns None. It reads no environment and no files.
• The variable follows the existing test-hook rule. It is not documented as a setting, and env_test_hook_boundary.py passes.

The only change in create.rs is in the restore path. It takes the fault, if one is armed, instead of calling prepare_sandbox. The result goes into the existing error handling.

Kind chart

• space-sandbox-manager-local sets the variable to a directory under the container's private /tmp (testHooks.restorePrepareFaultDir). A marker therefore never outlives its pod.
• Builder nodes do not get the variable (values-builder.yaml). An empty value renders no variable.
• No chart outside infra/space/kind names it. A new chart test checks this.

Smoke framework

• New capability RestoreRollbackControl in space_smoke/capabilities.py, with a Kind backend in adapters.py. It builds on main's RestoreStateRead and RestorePodRead:
  • arm_restore_prepare_fault, armed_restore_prepare_faults and disarm_restore_prepare_fault write, read and remove the marker. They run in the sandbox-manager container as its own user, on every sandbox worker, because the API chooses placement.
  • snapshot_artifact_rows reads every snapshots row of the sandbox, in any state.
  • sandbox_start_failures reads the pod's FailedCreatePodSandBox events. They are failure evidence only.
  • RestoreActivation now carries local_artifact_id from the runtime_activated contract event.
• New module space_smoke/restore_rollback.py holds the scenario steps and the verdict.

Scenario local.restore_rollback_late_retry_keeps_original_snapshot

Steps:

1. Create a sandbox, write a marker file, and pause it. Wait until the snapshot is portable, then record the one ready snapshot.
2. Arm the fault.
3. Connect. The restore prepare fails and rolls back. The connect fails. Check that exactly one marker file was consumed, so the fault caused the failure.
4. Check that the faulted pod still exists after the connect returned. Then keep it 30 s more, so kubelet keeps retrying the same attempt.
5. Delete the faulted pod (the teardown).
6. Connect again. Read the marker file and run a command.

The verdict follows space_smoke/AGENTS.md: pass or fail comes from state, not log lines. The scenario asserts:

• the sandbox still has exactly one snapshot row, the original, in any state;
• the sandbox's one active execution is newer than the faulted one;
• the runtime_activated contract event for that generation reports a snapshot boot of the original snapshot's local artifact (the scenario polls for the event and judges its fields);
• the marker file is intact and exec works.

Whether kubelet's late retries were refused, and whether the faulted generation activated, shows only in pod events and the sandbox-manager log. The scenario records these as failure evidence and does not assert on them. After the rolled-back placement, the successor pod carried no storage URI annotation on the bench, so the pod cannot name the snapshot. The activation event is the only exposed record of which artifact was restored.

The scenario is local.*, not ecw.*. The scenarios README reserves local.* for scenarios that need Kind-only control. ecw.* scenarios must hold on every backend, and this one needs the hook.

CI path filter

kind-pull-ci.yml now runs on any change under crates/libs/sbx/mgr/server/src/**. It used to run only on activation_support.rs. The restore, rollback and activation paths this scenario guards are spread across the server source.

How it was validated

Kind: passes with the fix, fails without it

I ran both builds on one local Kind bench, deployed through Tilt, after rebasing the stack on main at a5c67e5.

Build	Result	State	Evidence only
This stack (#583 plus this PR, aa200d2)	PASS in about 165 s	One snapshot row stayed. Generation 3 was the one active execution, and its activation reported a snapshot boot of the original local artifact. The marker file was intact and exec worked.	The first connect failed with 502. Every late retry of generation 2 was refused with FailedPrecondition: activation attempt ... was rolled back.
main (a5c67e5) plus only this PR's commit	FAIL: a new snapshot was published after the rolled-back restore; its late retry went live and its teardown suspended it again	Two ready snapshot rows, the second with a new local artifact.	Generation 2 activated (local_only) on kubelet's retry, and generation 3 activated the new artifact.

I ran the scenario at aa200d2. The head, eaa9870, changes only one docstring in space_smoke/restore_rollback.py. The failing build was a local branch only and was not pushed. Its smoke, chart and CI files are the same as aa200d2's.

Existing Kind smokes

Before the rebase, I ran CI's Kind shard 4 of 4 (scripts/tests/smoke-kind-typed.sh all 4/4) against this stack. I have not rerun it on the rebased stack. That shard holds the new scenario and the existing restore-heavy ones: ecw.pause_connect_race, local.encrypted_resuspend_preserves_cow, local.forced_drain_evacuates_sandbox and local.snapshot_reaper_reclaims_backed_fork_snapshot.

• ECW surface: 19 of 20 passed, including the new scenario.
• Native surface: 8 of 8 passed.
• ecw.cold_create_environment was inconclusive: cold setup requires the isolated local Kind API. Its setup only runs on a cluster whose context is exactly kind-space-local, and my bench is a named cluster. This is a limit of my bench, not a result of this change. CI runs it on kind-space-local.

I removed the test sandboxes, the run-scoped API keys and the fault markers afterwards.

Unit and repository checks

All of these were rerun on the rebased tree.

• cargo test -p pplx-space-sbx-mgr --lib restore_prepare_fault: 8 passed without test-hooks and 7 with it. One case runs only without the feature: it checks that no fault ever fires.
• cargo clippy -p pplx-space-sbx-mgr --tests -- -D warnings, with and without --features test-hooks, and cargo fmt --check: clean.
• cargo arch: passes.
• space_smoke/tests (pytest with the test dependency group, as ci/scripts/run-python-tests.sh runs it): 545 passed on the rebased tree. This includes the new test_restore_rollback_scenario.py, the inventory, sharding and protocol-conformance tests. The new unit tests check four things:
  • the state a broken build leaves fails with the named reason;
  • an activation event for the faulted generation is not judged by its presence;
  • a failure carries the logs as evidence, and the storage URI stays out of it;
  • the Python side uses the same variable and message as the Rust hook.
• Chart and hook guards: test_kind_restore_prepare_fault_chart.py (new), test_kind_known_egress_chart.py, test_kind_node_service_env.py, test_cluster_chart_validate.py, test_kind_telemetry_config.py and test_env_test_hook_boundary.py: 310 passed. helm lint passes.
• Source guards: env_test_hook_boundary.py, node_selector_extras_smoke.py and observability_vendor_boundary.py pass. thin_service_mains.py reports a finding in crates/apps/vol/mgr/svc/src/main.rs. This PR does not touch that file.

Open points

• The faulted connect takes about 120 s, because the API waits that long before it answers. That is most of the scenario's run time.
• Two call sites in sbx-mgr: skip teardown snapshot after a confirmed activation rollback #583 (the claim-time checks that only a race reaches) are still not reached by unit tests or by this scenario. See the mutation table in sbx-mgr: skip teardown snapshot after a confirmed activation rollback #583.
• Widening the CI path filter means more pull requests run the Kind suite.

AI disclosure

Perplexity Computer, an AI agent, wrote all of this PR: the Rust hook and its tests, the chart change, the smoke capability, the scenario and its tests, the CI filter change, and this description. It also ran the Kind validation. Nick Dujay directed the work and reviews it.