daemon: spurious re-stage and second, ungated reboot after mid-rollout image change #256

Summary

When a BootcNodePool's image is changed while a rollout is in progress, a node whose in-flight stage was cancelled can reboot twice: it reboots into the new image, comes back up, and the daemon then stages the same image again and reboots a second time. The second reboot is issued by the daemon on its own, with no controller involvement —it is not gated by the controller's cordon/drain/approval flow, so it disrupts the node outside the pool's maxUnavailable budget.

Observed as intermittent failures of TestMidRolloutImageChange (test/e2e/bootcnode_test.go), which asserts the second node reboots exactly once (afterCount-beforeCount == 1); failing runs show a delta of 2.

Evidence (from CI artifacts of failing runs)

Example run: https://github.com/bootc-dev/bootc-operator/actions/runs/39297483291 (e2e 1.37 job artifacts: k-get-events.txt, k-describe-bootcnodes.txt, controller logs).

• The affected node never booted the first update image (v1): its Rollback status entry is still the original base image, and its events go Staging v1 → Staging v2 after the revocation. So the extra reboot is not a case of booting the wrong image.
• Controller-manager logs show no controller action between the two reboots — the second reboot is entirely daemon-side.
• In every observed occurrence, the affected node is the one whose v1 stage was cancelled mid-flight (inflight.acquire → context cancel → SIGKILL of bootc stage) when the pool was patched to v2. The node that was already rebooting into v1 (clean, uncancelled stage) never double-reboots.

Suspected mechanism (under investigation)

The daemon decides "nothing to do" by comparing the desired digest against the booted image reported by the StatusWatcher (internal/daemon/reconciler.go:205-210; status filled by populateBootcFields via StatusWatcher.GetStatus, internal/daemon/watcher.go:44-55, which returns the cached status once started). If the daemon's view of the booted image is stale (still pre-reboot) after the node comes back, the early-return check misses, classifyAction falls through to actionStage, and the already-booted image is staged and rebooted again.

Open question: the daemon pod runs on the managed node, so a reboot restarts the process with an empty cache, and the first GetStatus does a live refresh. The plausible production variant is therefore fresh cache, stale data: the pod comes up during early boot and bootc status itself returns pre-reboot state, which is cached and acted on before fsnotify / the 5-minute poll correct it. This distinction matters for the fix — trusting the cache less would not help if the underlying read is stale
during early boot.

#251 (currently DNM) adds instrumentation to confirm this: a daemon-side log on every booted/desired mismatch, Idle-reason transition logging in the e2e test, andTestStaleCacheSpuriousRestage, a unit test demonstrating the spurious re-stage given a stale watcher cache.

Impact

• Operational: an unscheduled second reboot of a worker node, outside the rollout's drain/approval gating and maxUnavailable serialization.
• CI: intermittent e2e failures of TestMidRolloutImageChange.

Reproduction

Flaky. Rerun the bink e2e suite (make e2e); TestMidRolloutImageChange fails
intermittently when the pool image is patched while one node is mid-stage. CI is being
retriggered repeatedly on #251 to capture a reproduction with the instrumentation in
place.