[libvirt_manager] Add retries to controller SSH key inject tasks - #4057
[libvirt_manager] Add retries to controller SSH key inject tasks#4057skovili wants to merge 4 commits into
Conversation
Cloud-init may not finish writing authorized_keys before the inject tasks attempt SSH, causing "Permission denied (publickey)" on controller VMs. This is the same race the OCP SSH task already handles with retries. Related-Issue: #OSPRH-32065 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 32m 27s |
Ansible's retries/until mechanism only retries FAILED tasks, not UNREACHABLE ones. When controller-0 SSH is not yet ready (cloud-init race), the host is marked unreachable and removed from the play without retrying. Add ignore_unreachable: true so the task result includes unreachable status instead of ejecting the host, and extend the until condition to retry on both failed and unreachable states.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Replace ignore_unreachable + retries approach with wait_for_connection before the inject tasks. The previous approach failed because Ansible treats an "ignored unreachable" as task-complete, skipping the retry loop entirely. wait_for_connection has its own internal retry mechanism that handles UNREACHABLE hosts correctly, waiting up to 5 minutes for SSH to become ready after cloud-init completes. Related-Issue: #OSPRH-32065 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
29ecd4b to
cc5eb14
Compare
cc5eb14 to
4089772
Compare
4089772 to
4c5c723
Compare
|
Hello, there are currently two issues with this pull request. First, see that the check Include this signed-off-by line in your commit by including the -s flag when committing or amending a commit. Additionally, please target the Thanks! |
When the hypervisor is a remote host (not localhost), the ssh_jumper proxy config on the Zuul executor lacks the IdentityFile directive. This causes delegate_to connections to the controller VM to fail with Permission denied because the reproducer key only exists on the hypervisor. Replace delegate_to + ansible.builtin.copy with direct ssh/scp commands that run on the hypervisor, matching the pattern used by the existing OCP SSH access task. The hypervisor's SSH config has the IdentityFile, so these commands authenticate correctly. Related-Issue: #OSPRH-32065 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4c5c723 to
966f2dc
Compare
|
Taking another look at this pr, I notice it is broken up into 4 commits. When we merge prs, we don't squash commits together. If your changes don't need to be broken up into multiple commits, would you be able to combine them appropriately? Thank you! |
2 participants
Summary
retries: 5, delay: 30to both "Inject private key" and "Inject public key" tasks inmanage_vms.ymlretries: 5, delay: 60)authorized_keysbefore the inject task attempts SSH, causingPermission denied (publickey)on controller VMsRoot cause
SSHD starts independently of cloud-init on RHEL 9. The SSH port opens ~16s after VM boot, but cloud-init may not finish writing SSH keys until 30-60s post-boot. Without retries, the inject task fails immediately on first attempt.
Observed in testproject build
5ac3f032on devin07 (reused hypervisor, routable IPs through physical bridge), but this is a latent race condition that could affect any environment where cloud-init is slower than usual.Test plan
Depends-On:pointing to this PRRelated-Issue: #OSPRH-32065
🤖 Generated with Claude Code