EphemeralRunner pod-NotFound recovery recreates runner with used JIT config, producing zombie runners (session Conflict / deleted registration) after spot eviction

[vytautascirvinskas]

Checks

• ARC version: gha-runner-scale-set / gha-runner-scale-set-controller 0.14.2
• Kubernetes: v1.35.5 (AKS), runner pods on Azure Spot node pools (evictionPolicy=Delete)
• Runner image based on ghcr.io/actions/actions-runner:2.335.1, DinD mode

Bug description

When a runner pod disappears entirely (Azure Spot eviction deletes the VM, so the pod object returns NotFound), the EphemeralRunner controller immediately recreates a pod for the existing EphemeralRunner with the same runner name, runner ID, and (already-used) JIT configuration, without checking whether that ephemeral runner already had a job assigned.

The replacement pod can never do useful work. Observed in two variants (two incidents, a week apart):

Variant A — deleted registration (2026-07-23, runner ID 144231):

Runner connect error: Error: Conflict. Retrying until reconnected.
...
Registration 7b665912-035c-45b6-bbef-74e402e59364 was not found.
Failed to create a session. The runner registration has been deleted from the server.

The pod sat Running (looking healthy in every dashboard) for ~64 minutes until the controller treated its exit code 0 as successful completion.

Variant B — reconnect-then-jobless idle (2026-07-28, runner ID 146374):

11:12:19Z ERR  GitHubActionsService] POST request to https://broker.actions.githubusercontent.com/session failed. HTTP Status: Conflict
11:12:19Z      A session for this runner already exists.
11:12:19Z      Runner connect error: Error: Conflict. Retrying until reconnected.
   (30s retry loop; the dead pod's session was still live on the broker)
11:13:55Z INFO BrokerMessageListener] Session created.
11:13:55Z      Runner reconnected.
11:13:55Z      Listening for Jobs

The stale session expired, the replacement took it over — but its job had already been failed by GitHub ("The self-hosted runner lost communication with the server") and re-run elsewhere. The EphemeralRunner showed phase: Running with empty status.jobRequestId, idling on "Listening for Jobs" for ~35 minutes while genuine queued jobs waited for capacity (the zombie occupied a 4Gi/2cpu slot on the spot pool).

Timeline correlation in both cases: evictSpotVM events in the Azure activity log match the original pod's disappearance to the minute; the replacement pod is created seconds after the controller logs Ephemeral runner pod does not exist. Creating new ephemeral runner.

Why this happens (controller branch asymmetry)

In controllers/actions.github.com/ephemeralrunner_controller.go (0.14.2, still present on current master):

• If the pod exists with phase Failed and the EphemeralRunner has a job assigned, the controller correctly deletes the EphemeralRunner and removes the runner from GitHub (improved by Make restart pod more flexible to different failure scenarios #4340).
• If the pod object is already absent (NotFound — the spot-eviction case), the controller goes straight to creating a replacement pod for the existing EphemeralRunner. This branch does not check whether the runner already had a job, so the replacement is created with a consumed JIT config.

#4340 therefore does not cover this path, and #4148 describes the adjacent node-drain scenario.

Expected behavior

If an EphemeralRunner has (or had) an assigned job and its pod is NotFound, the controller should treat it like the pod-Failed-with-job case: delete the EphemeralRunner and remove the GitHub registration, letting the scale set create a fresh runner — instead of recreating a pod with a used JIT configuration.

Impact

• Dead runners look alive: dashboards/ArgoCD show a healthy Running pod with the same name as the runner that lost the job, which badly misleads incident triage (pod name is not a stable identity — only name+UID+runner ID is).
• Each spot eviction wastes a runner slot for 35–64 minutes on capacity-constrained pools.
• Jobs themselves are recoverable via re-run; the bug is the recovery path, not job loss.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[vytautascirvinskas]

Update after further observation on the same clusters (same day):

Correction: the zombie EphemeralRunners are not reaped at all. Variant B (reconnect-then-jobless) persisted 60+ minutes in a broker retry loop (Connection to Broker Server recreated ... Sleeping N seconds before retrying) with status.jobRequestId empty, and was only removed by manual kubectl delete ephemeralrunner. What we earlier took for self-cleanup was a pod recreation within the same EphemeralRunner.

They also multiply: each spot-eviction wave minted a new zombie — we accumulated three in one morning (runner IDs 146374 + two more, each showing the pod-created-minutes-after-ER + session-Conflict fingerprint). On a capacity-constrained pool each one holds a full runner slot while real jobs queue.

Reliable fingerprint for affected runners:

1. pod creationTimestamp several minutes newer than its EphemeralRunner (pod was recreated)
2. runner log contains the Session ... Conflict / A session for this runner already exists retry loop followed by Runner reconnected + Listening for Jobs with no job ever arriving
3. status.jobRequestId empty while phase: Running

Possibly the same root-cause family as #4155 (stuck ER after OOMKILL) and #4307 (stuck ER after job canceled/interrupted) — all three are cases where the runner loses its job without the pod reaching a clean observable Failed state, and the EphemeralRunner is never cleaned up.