AutoscalingRunnerSet wedges permanently after first teardown; a scale set name only ever registers once

[ubriggsl]

What happened

An AutoscalingRunnerSet loses its runner-scale-set-id annotation and then never re-registers. Every reconcile takes the teardown path and stops:

AutoscalingRunnerSet  Ephemeral runner set is outdated   {"autoscalingrunnerset":{"name":"lab-arc-runners"}}
AutoscalingRunnerSet  Deleting the listener
AutoscalingRunnerSet  Listener is deleted
AutoscalingRunnerSet  deleting ephemeral runner sets
AutoscalingRunnerSet  All ephemeral runner sets are deleted
AutoscalingRunnerSet  deleting runner scale set
<no further lines for this object>

The object parks at status.phase: Outdated with no AutoscalingListener, no EphemeralRunnerSet, and no runner-scale-set-id annotation. It never reaches the create path. Because no runner matches the label, queued jobs queue indefinitely rather than fail, so nothing surfaces an error — our first occurrence ran ~8 hours before it was noticed, and it was noticed by accident.

One error does surface, intermittently — the controller adding a finalizer to an EphemeralRunnerSet that has already been deleted underneath it:

ERROR EphemeralRunnerSet  Failed to update ephemeral runner set with finalizer added
  {"ephemeralrunnerset":{"name":"lab-arc-runners-74bc6"},
   "error":"ephemeralrunnersets.actions.github.com \"lab-arc-runners-74bc6\" not found"}

That looks like a create/delete race inside the reconcile: the ERS is created and then judged outdated and deleted within the same pass.

The reproducible pattern

A scale set name that has never been used registers on the first attempt and runs normally for 15–40 minutes. After its first teardown, that name never registers again.

scale set name	outcome
lab-arc-runners	wedged at its first teardown, never recovered
lab-arc-ci (fresh)	registered instantly (id 27), wedged ~25 min later, then refused
lab-ci-pool (fresh)	registered instantly (id 29), wedged ~15 min later
lab-arc-runners-xl	healthy all day — nothing ever tore it down

Deleting the AutoscalingRunnerSet and letting it be recreated restores the pool for another window, which is the only workaround we have found.

Ruled out by direct test

• Chart and controller version — rolled both 0.14.2 → 0.14.1; behaviour byte-identical. 0.14.2 had run fine for 8 days beforehand.
• The scale set name — three different names, same behaviour.
• The runner template — the wedging pool hand-rolled a dind sidecar; converged it onto the healthy pool's containerMode: dind shape; no change.
• minRunners — 24, 8 and 0 all wedge. 0 survives longer but still wedges.
• maxRunners — 2 and 30 both wedge.
• GitHub-side state — the ARC GitHub App reports zero live runner scale sets for the org via _apis/runtime/runnerscalesets, so there is no name collision or orphan.
• Argo CD — this is deployed by Argo CD, so we tested it thoroughly:
  • ignoreDifferences on .metadata.annotations."runner-scale-set-id" plus RespectIgnoreDifferences=true is present and live.
  • We then disabled automated sync entirely and synced exactly once by hand. It still wedged. Sync frequency is not the trigger.
  • resourceVersion on the ARS is stable between syncs; no write fight.
• Stale RBAC finalizers — cleared a leftover actions.github.com/cleanup-protection finalizer on the *-gha-rs-no-permission ServiceAccount; no change.
• Networking — DNS and TLS from inside the controller namespace to the exact Actions tenant host succeed in ~0.1s.

Environment

• gha-runner-scale-set + gha-runner-scale-set-controller 0.14.2, also reproduced on 0.14.1
• k3s v1.32, single cluster, org-level registration via GitHub App
• Deployed by Argo CD; minRunners: 0, maxRunners: 30, DinD via containerMode

What would help

Is there controller-side state tied to a scale set name that survives teardown and blocks re-registration? The fresh-name-works-once behaviour is the most specific clue we have, and it is what makes the failure permanent rather than transient.

[ubriggsl]

Additional data point that may narrow this: the pool can reach a registered-but-not-serving state, which looks healthy by every signal we had.

After a controller restart the AutoscalingRunnerSet held status.phase: Running with a runner-scale-set-id annotation and a live AutoscalingListener pod — stable across six samples over five minutes. Four workflow jobs targeting that scale set then sat queued for 5 minutes with zero EphemeralRunner pods created.

The listener logs only its periodic auth refresh during this window:

msg="refreshing token"
msg="getting access token for GitHub App auth"

and never logs the Handling initial session statistics / Calculated target runner count lines it emits when it is actually receiving assignments.

So there appear to be two distinct failure states, not one:

1. the wedge described above — phase: Outdated, no listener, no runner-scale-set-id
2. this one — everything reports healthy, listener connected, but no job is ever assigned

(2) is the more dangerous of the two operationally, because listener presence is the obvious thing to alert on and it reports green throughout.

[ubriggsl]

Resolved — and the root cause was not ARC. Closing, with the correction in full, because the misdiagnosis in this issue could cost someone else the same day it cost me.

The actual cause was a deprecated runner version. Our image was built on runner 2.334.0, and GitHub refuses it at the broker:

Current runner version: '2.334.0'
GET https://broker.actions.githubusercontent.com/message?...
An error occurred: Runner version v2.334.0 is deprecated and cannot receive messages.

The runner then exits "with terminated error, stop the service, no retry needed".

Why it presents as an ARC fault, which is the part worth documenting: every layer above the runner reports healthy. The scale set registers and holds a runner-scale-set-id, the listener connects and refreshes its token, the AutoscalingRunnerSet reports phase: Running, and totalAvailableJobs stays 0 because no runner can ever accept work. Jobs queue rather than fail, so nothing goes red anywhere. Rebuilding the image on a current runner (2.336.0) fixed it immediately — jobs now start in seconds.

Everything I listed above as "excluded by direct test" was excluded correctly; I was simply testing the wrong layer. Chart/controller version, scale-set name, template shape, minRunners, maxRunners, Argo CD sync policy — none was ever the variable.

The one diagnostic that would have short-circuited it: kubectl logs -n arc-runners <pod> -c runner. The pods are ephemeral and recycle fast, so it is easy to skip — but when a pool registers and serves nothing, the answer is inside the runner container, not in the controller or the CRD.

The secondary phase: Outdated wedging I described here is real and matches #4596, so I am closing this as a duplicate of that for the ARC-side behaviour. Sorry for the noise, and thanks to the maintainers.