AutoscalingRunnerSet remains permanently Outdated after runner template replacement

[omersadika]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.2

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install `gha-runner-scale-set-controller` and `gha-runner-scale-set` version `0.14.2` using Helm.
2. Configure the scale set with `minRunners: 0` and controller update strategy `immediate`.
3. Allow the scale set to create and run ephemeral runners.
4. Change the runner pod template in a way that changes its specification hash. For example, add labels and required pod anti-affinity.
5. Allow ARC to replace the outdated ephemeral runner set.
6. Observe ARC deleting the old listener and ephemeral runner resources.
7. Observe that the `AutoscalingRunnerSet` remains in `status.phase: Outdated` after cleanup.
8. Confirm that there is no `AutoscalingListener`, `EphemeralRunnerSet`, `EphemeralRunner`, or runner pod.
9. Observe that queued GitHub Actions jobs do not create replacement runners.

Describe the bug

After changing the runner pod template, ARC cleaned up the previous listener and ephemeral runner set but left the AutoscalingRunnerSet permanently in the Outdated phase.

The resulting resource had no runner-scale-set-id annotation but retained stale runner counts:

metadata:
  annotations:
    actions.github.com/runner-scale-set-name: example-scale-set
status:
  phase: Outdated
  currentRunners: 5
  pendingEphemeralRunners: 0
  runningEphemeralRunners: 5

There were actually no AutoscalingListener, EphemeralRunnerSet, EphemeralRunner, or runner pods.

The controller remained healthy and did not report an authentication, RBAC, or scheduling failure. Its reconciliation stopped after logging:

AutoscalingRunnerSet AutoscalingListener does not exist.
AutoscalingRunnerSet Ephemeral runner set is outdated
AutoscalingRunnerSet Deleting the listener
AutoscalingRunnerSet Listener is deleted
AutoscalingRunnerSet deleting ephemeral runner sets
AutoscalingRunnerSet All ephemeral runner sets are deleted
AutoscalingRunnerSet deleting runner scale set

Resetting only the status phase to Pending immediately recovered the deployment:

kubectl -n arc-runners patch \
  autoscalingrunnerset example-scale-set \
  --subresource=status \
  --type=merge \
  -p '{
    "status": {
      "phase": "Pending",
      "currentRunners": 0,
      "pendingEphemeralRunners": 0,
      "runningEphemeralRunners": 0,
      "failedEphemeralRunners": 0
    }
  }'

ARC then created a new GitHub runner scale set, listener, ephemeral runner set, and runners for the queued jobs. This confirmed that GitHub App authentication, Kubernetes RBAC, and runner scheduling were working.

The relevant reconciliation path appears to return successfully after cleanup without resetting the phase or explicitly requeueing:

https://github.com/actions/actions-runner-controller/blob/gha-runner-scale-set-0.14.2/controllers/actions.github.com/autoscalingrunnerset_controller.go#L246-L267

When the runner-scale-set-id annotation has already been removed, cleanup also returns successfully without changing the resource:

https://github.com/actions/actions-runner-controller/blob/gha-runner-scale-set-0.14.2/controllers/actions.github.com/autoscalingrunnerset_controller.go#L658-L706

This appears to leave the resource permanently in the Outdated phase without generating another event that would trigger creation of the replacement scale set.

Describe the expected behavior

After cleaning up an outdated runner set, ARC should transition the AutoscalingRunnerSet to a state that allows reconciliation to continue, or explicitly requeue the resource.

ARC should then create:

1. A replacement GitHub runner scale set
2. A new EphemeralRunnerSet
3. A new AutoscalingListener
4. New runners for queued jobs

The AutoscalingRunnerSet should not remain permanently in the Outdated phase after the previous listener, ephemeral runner resources, and runner-scale-set-id annotation have been removed.

The status runner counts should also be reset or updated so they do not report running runners when no corresponding EphemeralRunner resources or runner pods exist.

Additional Context

# gha-runner-scale-set-controller uses the chart defaults.
# The effective controller update strategy is "immediate".

# Sanitized gha-runner-scale-set values.yaml
githubConfigUrl: https://github.com/example-org
githubConfigSecret: example-github-config
runnerScaleSetName: example-scale-set

minRunners: 0
maxRunners: 8

controllerServiceAccount:
  name: arc-controller
  namespace: arc-system

template:
  metadata:
    labels:
      example.com/runner-scale-set: example-scale-set
  spec:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                example.com/runner-scale-set: example-scale-set
            topologyKey: kubernetes.io/hostname

    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command:
          - /home/runner/run.sh
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work

    volumes:
      - name: work
        emptyDir: {}

Controller Logs

2026-08-07T12:45:15Z INFO AutoscalingRunnerSet AutoscalingListener does not exist. {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:45:15Z INFO AutoscalingRunnerSet Ephemeral runner set is outdated {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:45:15Z INFO AutoscalingRunnerSet Deleting the listener {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:45:15Z INFO AutoscalingRunnerSet Listener is deleted {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:45:15Z INFO AutoscalingRunnerSet deleting ephemeral runner sets {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:45:15Z INFO AutoscalingRunnerSet All ephemeral runner sets are deleted {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:45:15Z INFO AutoscalingRunnerSet deleting runner scale set {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}

# No further reconciliation occurred for approximately 12 minutes.
# The following lines appeared immediately after manually resetting status.phase to Pending.

2026-08-07T12:57:49Z INFO AutoscalingRunnerSet Runner scale set id annotation does not exist. Creating a new runner scale set. {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:57:50Z INFO AutoscalingRunnerSet Created/Reused a runner scale set {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"},"id":3,"runnerGroupName":"Default"}
2026-08-07T12:57:50Z INFO AutoscalingRunnerSet Updated with runner scale set ID, name and runner group name as an annotation {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"},"id":3,"name":"example-scale-set","runnerGroupName":"Default"}
2026-08-07T12:57:50Z INFO AutoscalingRunnerSet Latest runner set does not exist. Creating a new runner set. {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:57:50Z INFO AutoscalingRunnerSet Creating a new EphemeralRunnerSet resource {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"}}
2026-08-07T12:57:50Z INFO AutoscalingRunnerSet Created a new AutoscalingListener resource {"version":"0.14.2","autoscalingrunnerset":{"name":"example-scale-set","namespace":"arc-runners"},"name":"example-scale-set-listener","namespace":"arc-system"}
2026-08-07T12:57:53Z INFO EphemeralRunnerSet Creating new ephemeral runners (scale up) {"version":"0.14.2","ephemeralrunnerset":{"name":"example-ephemeral-runner-set","namespace":"arc-runners"},"count":3}
2026-08-07T12:57:54Z INFO EphemeralRunner Created ephemeral runner pod {"version":"0.14.2","ephemeralrunner":{"name":"example-runner-1","namespace":"arc-runners"},"runnerScaleSetId":3,"configUrl":"https://github.com/example-org"}
2026-08-07T12:57:54Z INFO EphemeralRunner Created ephemeral runner pod {"version":"0.14.2","ephemeralrunner":{"name":"example-runner-2","namespace":"arc-runners"},"runnerScaleSetId":3,"configUrl":"https://github.com/example-org"}
2026-08-07T12:57:55Z INFO EphemeralRunner Created ephemeral runner pod {"version":"0.14.2","ephemeralrunner":{"name":"example-runner-3","namespace":"arc-runners"},"runnerScaleSetId":3,"configUrl":"https://github.com/example-org"}

Runner Pod Logs

# No runner pod logs were available during the failure because no runner pods existed.

$ kubectl -n arc-runners get pods
No resources found in arc-runners namespace.

# The AutoscalingRunnerSet status incorrectly reported five running runners:
#
# status:
#   phase: Outdated
#   currentRunners: 5
#   pendingEphemeralRunners: 0
#   runningEphemeralRunners: 5
#
# There were no corresponding EphemeralRunner resources or Kubernetes pods.
#
# After manually resetting status.phase to Pending, ARC immediately created
# replacement runner pods. Those runners registered successfully and began
# executing the queued jobs, so there was no runner-container error to report.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[carolinapc]

Independent reproduction with a much tighter repro window than the original report — same terminal state, but no config/template change and no prior corrupted history involved at all.

Environment: controller/chart 0.14.2, GKE, updateStrategy: eventual (we deliberately switched away from the chart default immediate suspecting it was required to trigger this — it made no difference, see below).

Repro, minimal:

1. AutoscalingRunnerSet with minRunners: 0, freshly created via pulumi destroy + redeploy (brand new object, generation: 1, no prior stuck state, no annotations carried over).
2. Confirmed healthy: listener Running, EphemeralRunnerSet created.
3. Triggered a single real GitHub Actions job against it. Runner picks it up, job completes successfully.
4. EphemeralRunnerSet scales back down to 0 replicas (normal minRunners: 0 behavior).
5. Within the same second, the controller logs:

   AutoscalingRunnerSet AutoscalingListener does not exist.
   AutoscalingRunnerSet Ephemeral runner set is outdated
   AutoscalingRunnerSet Deleting the listener
   ...
   AutoscalingRunnerSet Deleted the runner scale set from Actions service
   

   ...twice in immediate succession, then silence. Terminal state matches this issue exactly: status.phase: Outdated, no listener, no EphemeralRunnerSet, no runner-scale-set-id annotation, zero pods.

This happened again identically on the same object after we recovered it with the status.phase reset workaround, ran one more job through it, and it scaled back to zero — same immediate flip, same terminal state.

We also confirmed this is independent of updateStrategy: we switched our controller from the chart default immediate to eventual specifically hoping to avoid this (both #4595 and #4596 mention immediate in their repro steps), and saw no change — objects with active job traffic still flip to Outdated on essentially every scale-to-zero transition, while idle objects with no job traffic never enter the outdated branch at all and never hit this.

Given how tight this repro is (fresh object → one successful job → immediate break on scale-to-zero, updateStrategy irrelevant), the trigger condition may be narrower than "a config change coincides with an Actions outage" — it looks like it can also just be the ordinary 1→0 EphemeralRunnerSet scale-down transition itself, at least in our environment. Happy to share full controller logs (debug level) if useful.

[lmbaschiera]

@carolinapc - Hit the same issue, and it seems to be caused by the runner version you're running. Nothing fixed it until we upgraded from 2.334.0 to 2.336.0, since every other job spun up was being marked as Outdated by the controller (We spent HOURS debugging this, and went into inspecting the actual controller code to figure out what was going on, since Debug logs were not helpful at all) - Hope this helps.

[carolinapc]

@lmbaschiera I've upgraded to from 2.334.0 to the latest 2.336.0 and it's working now. Thank you

[abravonunez]

Confirming this fixed it for us too. Same symptoms as reported here and in #4595: AutoscalingRunnerSet permanently stuck in Outdated phase, runner-scale-set-id annotation removed, listener/EphemeralRunnerSet/pods all torn down with no error logs and no further reconciliation. Reproduced identically across controller versions 0.14.0, 0.14.1, and 0.14.2 — so it's not version-specific to the controller itself.

Upgraded the runner agent from 2.334.0 → 2.336.0 and it's now stable (previously it never survived more than ~30s before flipping back to Outdated).

Possibly relevant context: GitHub recently resumed minimum-version enforcement for self-hosted runners, which requires staying within 30 days of the latest release to keep executing jobs, not just meeting the 2.329.0 floor to register:
https://github.blog/changelog/2026-06-12-github-actions-minimum-version-enforcement-timeline-for-self-hosted-runners/

If the Actions service is silently invalidating registrations for agents that fall outside that rolling window, that could explain why the controller's cleanup path runs successfully but the recreate path never stabilizes — worth checking if this correlates with agent age rather than a purely client-side hash bug. Hope this helps narrow it down.