Skip to content

gha-runner-scale-set 0.14.2: deprecated-runner exit code 7 triggers scale-set delete/recreate loop; unbounded DeleteRunnerScaleSet call can wedge the controller permanently #4600

Description

@MuncleUscles

Checks

  • Chart/controller version: gha-runner-scale-set-controller 0.14.2 (ghcr.io/actions/gha-runner-scale-set-controller:0.14.2), EKS 1.35, org-scoped scale set with runnerGroup, custom runner image based on ghcr.io/actions/actions-runner:2.334.0.

What happened (production incident, 2026-08-10)

  1. GitHub's rolling minimum runner version passed our pinned actions/runner 2.334.0. The service rejects the runner and the container exits with code 7, which ephemeralrunner_controller maps to phase Outdated ("Runner is deprecated").
  2. The Outdated phase escalates: EphemeralRunner → EphemeralRunnerSet (updateStatus: any outdated child ⇒ ERS Outdated) → AutoscalingRunnerSet (outdated at line ~231) → cleanUpResources tears down the listener, all runner sets, and deletes the GitHub-side runner scale set — then recreates everything, pulls the same stale image, and loops. A version-deadline event thus produces an infinite delete/recreate storm of the scale-set object itself.
  3. During that storm, one DeleteRunnerScaleSet call never returned. The Actions client sets no request deadline and the reconcile context is unbounded, so the single AutoscalingRunnerSet worker blocked for hours: pod Ready, zero log output, healthz green, no listener, 0 runners. Liveness cannot detect it. Restarting re-enters the same stuck delete (phase Outdated is persisted in status; its only exit is a successful cleanup).
  4. We additionally observed the stuck call did not honor HTTP(S)_PROXY env on the controller (other calls from the same manager, e.g. JIT-config, did traverse our egress proxy), so a proxy-enforced timeout could not bound it either.

Asks

  1. Deadlines on all Actions service calls (or reconcile-scoped timeouts) so a stalled call errors and requeues instead of starving the worker forever.
  2. Deprecated runner (exit 7) should not delete the GitHub-side scale set — tearing down and recreating the scale set cannot fix a stale image and amplifies API load; surfacing a terminal condition/event ("runner image below minimum version") would make the failure diagnosable in minutes instead of hours.
  3. A way out of persisted AutoscalingRunnerSetPhaseOutdated that does not require the GitHub-side delete to succeed.
  4. Reconcile-progress signal in /healthz (or a metric) so a wedged worker is restartable by liveness.

Reproduction sketch

Pin a runner image below the service minimum in a scale set with minRunners >= 1 and watch the Outdated teardown loop; any long-stalling DeleteRunnerScaleSet response then wedges the controller permanently.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions