Skip to content

DNM e2e: fix flaky TestMidRolloutImageChange boot count assertion - #251

Open
alicefr wants to merge 2 commits into
bootc-dev:mainfrom
alicefr:fix-midrollout
Open

alicefr wants to merge 2 commits into
bootc-dev:mainfrom
alicefr:fix-midrollout

Conversation

@alicefr

@alicefr alicefr commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Debugging TestMidRolloutImageChange test

@alicefr
alicefr requested a review from ptalgulk01 October 22, 2026 07:42
@ptalgulk01

ptalgulk01 commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks for looking into this! The flake is real, but the CI logs from the two recent failures (Oct 16, Oct 19) from other PR.

  • The poll caught the first node rebooting both times (78s/114s after the v1 patch, v2 switched 13–30ms later) — it can't miss it, since Rebooting lasts minutes and we poll every 2s.
  • The failing "other" node never booted v1 (rollback entry = original image, events show Staging v1 → Staging v2). The revocation path works fine.
  • What actually happened: after rebooting into v2, the node's daemon staged v2 again and rebooted again — same image, second reboot, with no controller involvement in between (controller logs confirm no new drain/approval). Boot count went 2→4.

So this looks like a real daemon bug, not a test race — likely fallout from the cancelled v1 stage (the first node, which staged v1 cleanly, never double-reboots).

@alicefr alicefr changed the title e2e: fix flaky TestMidRolloutImageChange boot count assertion DNM e2e: fix flaky TestMidRolloutImageChange boot count assertion Oct 22, 2026
Add temporary debug logging to investigate the spurious re-stage
after a cancelled bootc switch. The daemon logs the exact booted
and desired digests when InfoMatchesDigest returns false, and the
test logs BootcNode status at each state transition plus raw
bootc status on the other node after convergence.

Assisted-by: AI

@HarshwardhanPatil07 HarshwardhanPatil07 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alicefr

alicefr commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

some ci unit test seems to be failing here https://github.com/bootc-dev/bootc-operator/actions/runs/39783566535/job/119840734257?pr=251

That's not important. It is a unit test that I added to try one scenarios, but it won't be merged if we don't confirm that's the real cause

@alicefr

alicefr commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Trying to reproduce the flaky test by triggering multiple times the ci

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants