Checks
- Chart/controller version: gha-runner-scale-set-controller 0.14.2 (
ghcr.io/actions/gha-runner-scale-set-controller:0.14.2), EKS 1.35, org-scoped scale set with runnerGroup, custom runner image based on ghcr.io/actions/actions-runner:2.334.0.
What happened (production incident, 2026-08-10)
- GitHub's rolling minimum runner version passed our pinned
actions/runner 2.334.0. The service rejects the runner and the container exits with code 7, which ephemeralrunner_controller maps to phase Outdated ("Runner is deprecated").
- The Outdated phase escalates: EphemeralRunner → EphemeralRunnerSet (
updateStatus: any outdated child ⇒ ERS Outdated) → AutoscalingRunnerSet (outdated at line ~231) → cleanUpResources tears down the listener, all runner sets, and deletes the GitHub-side runner scale set — then recreates everything, pulls the same stale image, and loops. A version-deadline event thus produces an infinite delete/recreate storm of the scale-set object itself.
- During that storm, one
DeleteRunnerScaleSet call never returned. The Actions client sets no request deadline and the reconcile context is unbounded, so the single AutoscalingRunnerSet worker blocked for hours: pod Ready, zero log output, healthz green, no listener, 0 runners. Liveness cannot detect it. Restarting re-enters the same stuck delete (phase Outdated is persisted in status; its only exit is a successful cleanup).
- We additionally observed the stuck call did not honor
HTTP(S)_PROXY env on the controller (other calls from the same manager, e.g. JIT-config, did traverse our egress proxy), so a proxy-enforced timeout could not bound it either.
Asks
- Deadlines on all Actions service calls (or reconcile-scoped timeouts) so a stalled call errors and requeues instead of starving the worker forever.
- Deprecated runner (exit 7) should not delete the GitHub-side scale set — tearing down and recreating the scale set cannot fix a stale image and amplifies API load; surfacing a terminal condition/event ("runner image below minimum version") would make the failure diagnosable in minutes instead of hours.
- A way out of persisted
AutoscalingRunnerSetPhaseOutdated that does not require the GitHub-side delete to succeed.
- Reconcile-progress signal in
/healthz (or a metric) so a wedged worker is restartable by liveness.
Reproduction sketch
Pin a runner image below the service minimum in a scale set with minRunners >= 1 and watch the Outdated teardown loop; any long-stalling DeleteRunnerScaleSet response then wedges the controller permanently.
Checks
ghcr.io/actions/gha-runner-scale-set-controller:0.14.2), EKS 1.35, org-scoped scale set withrunnerGroup, custom runner image based onghcr.io/actions/actions-runner:2.334.0.What happened (production incident, 2026-08-10)
actions/runner2.334.0. The service rejects the runner and the container exits with code 7, whichephemeralrunner_controllermaps to phaseOutdated("Runner is deprecated").updateStatus: any outdated child ⇒ ERSOutdated) → AutoscalingRunnerSet (outdatedat line ~231) →cleanUpResourcestears down the listener, all runner sets, and deletes the GitHub-side runner scale set — then recreates everything, pulls the same stale image, and loops. A version-deadline event thus produces an infinite delete/recreate storm of the scale-set object itself.DeleteRunnerScaleSetcall never returned. The Actions client sets no request deadline and the reconcile context is unbounded, so the single AutoscalingRunnerSet worker blocked for hours: podReady, zero log output, healthz green, no listener, 0 runners. Liveness cannot detect it. Restarting re-enters the same stuck delete (phaseOutdatedis persisted in status; its only exit is a successful cleanup).HTTP(S)_PROXYenv on the controller (other calls from the same manager, e.g. JIT-config, did traverse our egress proxy), so a proxy-enforced timeout could not bound it either.Asks
AutoscalingRunnerSetPhaseOutdatedthat does not require the GitHub-side delete to succeed./healthz(or a metric) so a wedged worker is restartable by liveness.Reproduction sketch
Pin a runner image below the service minimum in a scale set with
minRunners >= 1and watch the Outdated teardown loop; any long-stallingDeleteRunnerScaleSetresponse then wedges the controller permanently.