[ac-pplx]

The gap

Every Kind Rust binary is built with test-hooks (docker/Dockerfile.rust-build-kind's feature lists), so each environment variable read through pplx_space_common_core::env::test_hook/test_hook_var takes effect, and the Kind charts set several on every run. Nothing exercised the platform with binaries built without the feature, or checked that a bench claiming such binaries actually runs them.

Inventory: the test hooks the Kind charts set

Hook	Reader	Code default	Set by the Kind charts	Scenarios that depend on it
SPACE_SBX_GC_TARGET_FRACTION, _INTERVAL_SECS, _GRACE_SECS, _IDLE_SECS	sbx-mgr snapshot_gc::GcConfig::from_env	0.40 / 300 s / 1800 s / 86400 s	sandbox-manager chart, every run (workers): 0 / 15 / 60 / 120	local.snapshot_reaper_reclaims_backed_fork_snapshot
SPACE_TEMPLATE_GC_INTERVAL_SECS, SPACE_TEMPLATE_GC_GRACE_SECS	template-mgr eviction::GcConfig	300 s / 1800 s	template-manager chart, every run: 15 / 60	local.template_cache_lru_eviction_reclaims
SPACE_TEMPLATE_PREFETCH_LATEST_N	template-mgr prefetch (LATEST_N_TEST_HOOK)	5	template-manager chart, every run: 1	same
SPACE_NODE_DISK_ADMISSION_ALLOCATION_RESERVATION_BYTES	common-core CacheDiskAdmissionConfig::from_env	8 GiB	node-service library, every run: 1 GiB on node-manager, sandbox-manager, volume-manager, build-manager	none (the lane passes without it)
SPACE_LOOP_DEVICE_TARGET	node-mgr loop_devices::override_from_env	derived from the slot budget	node-manager chart, only when the bring-up's env sets it (Tiltfile/bootstrap.sh pass-through)	none

The five scenarios the lane skips also stage hooks of their own at run time (the retention sweeper's dry-run, tick and minimum age; the activation-admission burst); see Skipped in the lane.

The change

SPACE_KIND_RUST_BUILD=release (new infra/space/kind/lib/rust-build.sh; the default is test-hooks) is read by the Tiltfile, bootstrap.sh, build-rust-artifacts.sh and the smoke bring-up.

• Build. The two test-hook lists in Dockerfile.rust-build-kind become build args (SPACE_KIND_TEST_HOOK_FEATURES, SPACE_KIND_GUEST_TEST_HOOK_FEATURES). Both artifact modes read them: Docker passes them empty, and incremental leaves them out. The lane builds every Kind service with no crate's test-hooks feature enabled. That requires leaving the test-support member pplx-space-db-test-fixtures out of the package selection (SPACE_KIND_CARGO_EXCLUDE): the host builds pass --workspace --exclude pplx-space-db-test-fixtures, the Docker cook names every other member with -p (cargo chef cook has no --exclude; the planner derives the list from cargo metadata), and the guest cook selects the guest build's own packages, so the cook and the build resolve the same features. The lane also sets CARGO_PROFILE_RELEASE_LTO=thin, refuses SPACE_KIND_RUST_PROFILE=dev-release, and never exports to the shared layer cache: the guard runs as soon as the build mode is resolved, before chef-base, the health probe or either artifact mode is built.
• Image labels. The artifact image carries space.pplx.net/kind-rust-build, which prebuilt mode checks, and runtime-image-cache.sh copies that label onto every runtime image built from the artifact image (its SPACE_RUST_BUILD_IMAGE build arg). The incremental reuse key includes the build.
• Charts. Every chart is rendered with rustBuild=<build>. In release, the space-node-service library's rustBuild helper gates the reservation hook, sandbox-manager's four reaper hooks, template-manager's three cache hooks and node-manager's SPACE_LOOP_DEVICE_TARGET. The release-read settings next to them stay: SPACE_SBX_GC_ENABLED, _ENGAGE_FRACTION, SPACE_TEMPLATE_CACHE_BUDGET_BYTES and _MIN_AVAILABLE_BYTES. Every Rust workload's pod template (api, admin-api, internal-api, credmgr-authority, key broker, data-plane, cluster-controller, every node service and the operator shell) carries the space.pplx.net/kind-rust-build label, so a change of build rolls all of them. The label renders only when rustBuild is set, so the space-cluster umbrella chart, which never sets it, renders exactly as on main.
• What runs. The new infra/space/kind/scripts/deployed-rust-build.py decides the bench's build from what runs: for every running container of a SPACE Rust image (space-api, space-data-plane, space-cluster-controller, space-worker, space-kind-key-broker) it reads the label of the image that container runs, on its node (crictl inspecti), and requires every container, and each pod's rendered label, to agree. --wait first waits for every Deployment and DaemonSet that runs one of those images to roll out, and --pods prints the per-pod evidence. bootstrap.sh refuses to render the charts when the artifact image or a Rust runtime :local image carries the other build (as --skip-images can leave them), and fails unless every container runs the requested build after the rollouts. The smoke bring-up (bootstrap-kind-stack.sh) uses the same reader before deciding whether to restart Tilt, and after Tilt settles it waits for every Rust rollout and checks again. The typed smoke's _apply_rust_build uses it too.
• Scenarios. A new registry field, test_hooks= (naming the hooks a scenario needs), is set on the five scenarios that cannot pass without one. On a release bench the suite skips those five and prints them as SKIPPED. When SPACE_KIND_RUST_BUILD is set and the bench runs the other build (or no single build), the suite fails. The bring-up restarts Tilt into the requested build, or refuses under SKIP_BOOTSTRAP=1, mirroring the storage-credentials mode.
• CI. The new .github/workflows/kind-release-ci.yml (workflow_dispatch only) calls kind-ci.yml with the new rust_build: release input. The producer builds the release image in Docker mode; the four shards verify its label, then run the baseline and their typed partition. The default caller passes nothing, and SPACE_KIND_RUST_BUILD stays empty there, so the required lane runs as before.

Why this is faithful. The lane compiles the same bins and features as the default, inside the same chef-base builder, with thin LTO, and selects packages so that no crate's test-hooks feature is on. The charts render only settings a binary without the feature reads. The default lane builds the same binaries and renders the same env; its Rust pods only gain the label.

Skipped in the lane:

• local.template_cache_lru_eviction_reclaims: the code-default GC grace is 1800 s, past its 900 s budget.
• local.snapshot_reaper_reclaims_backed_fork_snapshot: the code-default idle threshold is a day.
• local.retention_claim_finish_forward and local.retention_restore_source_guard: the code-default tick is 300 s and min age is an hour, against an 80 s budget sized from the 5 s tick they stage.
• local.rate_limiters_enforce_and_fail_open: draining a burst of 60 would take sixty real boots.

What still differs

• The lane always runs the code-default reservation (8 GiB). The 40 GiB cache admitted it for the whole suite, including the concurrent scenarios, so it is not a hard limit. The MIN_AVAILABLE_BYTES floor is a setting the binaries read without the feature, and stays at 1 GiB.
• The reader covers the long-running Rust workloads. The seed template's guest binaries come from the same artifact image through build-seed.sh; its publish Job's pod does not stay running, so the reader does not inspect it.
• Everything else in the "Differences" table is unchanged.

README "Differences" rows changed (and nothing else in the table):

• "Release binaries read no test hooks": the left column is unchanged from main; the right column adds the lane.
• "8 GiB disk-admission reservation": the lane runs the code-default reservation.
• "Release binaries, built without test-hooks": the lane builds with thin LTO and no crate's test-hooks feature.

Also added: a new ## Release-build lane section, one sentence in the Tilt paragraph, and one line in the Kind AGENTS.md.

Validation

The Kind runs below validated 0fe935d on base b95b6ab, except run 6, which validated the final head 9d7999e on base be16af3 (the same patch rebased; git range-diff shows it unchanged). Settings matched CI: SPACE_KIND_TELEMETRY_MODE=otel, DATA_PLANE_URL=http://127.0.0.1:8090, and the admin token file and cold-receipts dir as in kind-ci.yml. The health gate (Tilt quiet, no ReplicaSet younger than 180 s, nodes Ready and published, 120 s settle) ran before the typed suite. Provenance is by binary sha256: for nine services (api, internal-api, data-plane, cluster-controller, credmgr-authority, key broker, sandbox-manager, template-manager, node-manager), the binary in a running pod matched the one in the staged build, and the reader's per-pod output (deployed-rust-build.py --pods) covered all 50 running Rust containers.

Release build time, Docker mode (the CI producer's path), cold: build cache pruned, chef-base rebuilt

Stage	Before (4b86967's Dockerfile)	After
Host cook	69.0 s	67.4 s
Guest cook	18.3 s	19.8 s
Host build	336.3 s, 60 third-party crates recompiled	339.8 s, 0 recompiled
Guest build	36.0 s, 115 recompiled	26.6 s, 0 recompiled
Supervisor build	29.9 s, 32 recompiled	33.0 s, 50 recompiled
Total	571 s	570 s

The cook now resolves the same features as the host and guest builds, so those builds recompile no dependency. Wall time didn't change: the host build's critical path is the workspace crates and the thin-LTO links. The supervisor is a separate invocation with its own feature set, as in the default lane, so it still recompiles some dependencies.

Results

Check	Result
Lane feature set	Pass. cargo tree -e normal,build,features over the lane's selection and features shows no crate with test-hooks enabled, for the host, guest and supervisor invocations. In the release binaries, the hook-only names SPACE_SBX_GC_IDLE_SECS, SPACE_TEMPLATE_GC_INTERVAL_SECS and SPACE_NODE_DISK_ADMISSION_ALLOCATION_RESERVATION_BYTES are absent; settings read without the feature, such as SPACE_SBX_GC_ENGAGE_FRACTION, remain.
Chart renders	Pass. The space-cluster umbrella (values.example.yaml) renders byte-identical to main. The first round's head added 31 kind-rust-build: test-hooks labels there. Every changed Kind chart in default mode differs from main only by the build label. Release mode drops exactly the eight chart hooks, plus SPACE_LOOP_DEVICE_TARGET when one is passed.
1. Fresh default cluster (smoke bring-up), e2e	Pass ([7/7] PASS round-trip ok). 50 of 50 Rust containers run test-hooks images, rendered test-hooks.
2. Reuse via bootstrap.sh	Starting from that cluster under bootstrap.sh (default, :local images): SPACE_KIND_RUST_BUILD=release --skip-images refused before rendering any chart (space-rust-build-kind:local is the test-hooks Rust build …), and no workload's generation changed. SPACE_KIND_RUST_BUILD=release with images rolled all 25 Rust Deployments and DaemonSets: 50 of 50 containers run release images, and the sampled binaries match the release build and differ from the default one. Reverse: default --skip-images refused the same way, then default bootstrap.sh rolled back to 50 of 50 test-hooks, with binaries byte-identical to run 1's.
3. Reuse via the smoke bring-up, to release	Default cluster, no Tilt, API up: SPACE_KIND_RUST_BUILD=release smoke-kind-e2e.sh started Tilt in release, waited for all 25 Rust rollouts, then read 50 of 50 release containers; e2e passed. On that bench, SKIP_BOOTSTRAP=1 with the default request was refused (runs the release Rust build (read from its Rust containers' images)). The typed suite's gate, run against it, kept 78 of 83 ECW scenarios with release (the five SKIPPED) and returned the mismatch error with test-hooks.
4. Reuse via the smoke bring-up, back to default	With Tilt live in release, smoke-kind-e2e.sh (default) restarted Tilt, waited for all 25 rollouts, then read 50 of 50 test-hooks containers, with binaries identical to run 1's; e2e passed.
5. Fresh release cluster, e2e and the release-safe set	e2e passed; 50 of 50 containers release; binaries match the release build. smoke-kind-typed.sh all, 73 min: ECW 78 passed / 0 failed, including local.ecw_template_build_from_oci_boots, local.concurrent_restore_converges_to_one_execution and local.cold_sandbox_eviction_wake_round_trip. Native: 25 passed / 1 failed, including native_contract.zero_vcpu_admission. The five hook scenarios printed as SKIPPED. The failure is native.m1_generated_sdk_conformance (AttributeError: 'str' object has no attribute 'to_dict'); it failed the same way on a rerun. PRE-EXISTING: Main CI fails shard 4 with the same error on b95b6ab.
6. Final head, release	Tilt rebuilt the rebased tree in release. The smoke bring-up reused the live stack, waited for all 25 rollouts, then read 50 of 50 release containers; e2e passed.
Harness tests	Pass: space_smoke and test_kind_registry_cache.py (554 passed, 1 skipped), test-kind-rust-artifacts-{mode,reuse,prebuilt}.sh, test-kind-port-forward-parity.sh, test-kind-stale-tilt-recovery.sh, test-kind-tombstone-sweep.sh.
Rust checks	Not applicable: no crate changed.
CI	On 9d7999e, every check passed except Kind shard 4, which fails only native.m1_generated_sdk_conformance with the same AttributeError. PRE-EXISTING: Main CI fails shard 4 the same way on b95b6ab and be16af3, and passed on 188c1a3. Shards 1–3 passed, as did the default-lane producer build. The new dispatch workflow can be dispatched only after it exists on the default branch, so it is not exercised here.

No new tests: the change is limited to charts, scripts, the smoke harness and CI, so the mismatch, refusal and skip paths were exercised by the real runs above. Two existing harness tests' fixtures now stub the new reader instead of the old DaemonSet read.

Bench disk. Each Tilt rebuild cycle on this box left about 11 GB of BuildKit cache on a 49 GB Docker root. Free space fell to 4 GB during one cycle's image builds (no build failed), and the cache was pruned between runs.

Nightly schedule? Worth it once the lane has a few green dispatches. The lane is the only check of binaries built without the feature, and a regression that appears only without hooks (a default the charts override) would otherwise go unseen. One nightly run costs a producer and four shards. Not added here.

Review follow-up

• The PR text, README, Dockerfile comments and commit message described more than the lane: fixed. They now describe only the lane's mechanism (no crate's test-hooks feature enabled, which requires excluding the test-support member). The "Release binaries read no test hooks" row's left column is back to main's wording, and the inventory lists only hooks the Kind charts set.

• Switching a reused cluster could leave the control plane on test-hook binaries while the check passed: fixed. The build now reaches every Rust workload's pod template, the bring-ups wait for the replacement pods, and the build is read from each running container's image on its node and checked against the pod's render. bootstrap.sh also refuses images of the other build before rendering.

• The release lane could still export chef-base and the health probe, and incremental mode never applied the guard: fixed. Cache exports are now disabled as soon as release mode is resolved.

• The release cook selected every member while the build excluded the fixtures crate: fixed. The host cook now gets the build's selection as -p, and the guest cook its build's packages. Dependencies recompiled in the build drop to zero; the wall time didn't change (see Validation).

• SPACE_LOOP_DEVICE_TARGET still rendered in release mode: fixed by gating it like the other hooks. The chart value feeds only node-manager, which reads it through test_hook_var; the node image's staging reads its own env file, so nothing needed splitting.

• Stale comments (the adapter's "every workload is labelled", the Tiltfile's and build-images.sh's "LTO off"): fixed. No tests were added for the harness's mismatch and skip paths, per the repository's test policy for harness changes; they are exercised by real runs instead.

• Rebased onto main after the Rust input-key reuse landed: the release lane's input key now names its build (rust-build release), so it can never match a published test-hooks image, while the default build's key is unchanged; Kind CI skips the reuse of main's artifact image for a release run; prebuilt mode checks both the build label and, for another commit, the input key. Checked locally: the default and explicit test-hooks keys are equal and the release key differs.

• Rebased onto main after kind: run the reconcile-queue GC against a fake EC2 endpoint #698 and the per-scenario recheck (kind_stack_recheck_for_scenario) landed: the SKIP_BOOTSTRAP=1 path checks the Rust build first, then the shared storage-mode refusal and the reconcile-queue GC wiring. The recheck's comment now names the Rust build check beside image provenance as one only a Tilt build changes, so it stays out of the per-scenario recheck.

Rollback

Revert the commit. The default lane renders the same env and builds the same binaries. Clusters brought up with the revert lose only the space.pplx.net/kind-rust-build pod label, which rolls the Rust pods once.

---

🤖 Generated with Claude Code

[nikhilgupta-pplx]

The release lane is opt-in, and the build/cache isolation looks sound. Default chart renders only gain build labels; the umbrella render is unchanged. The 45-file scope is consistent with carrying the build through artifacts, workloads and smoke selection.

Validated 572 Python tests, the focused artifact/reuse/bootstrap shell tests, and chart renders against the merge base. No code blockers. Required CI still needs to finish; run the new dispatch lane after it lands before treating it as CI-proven.