Skip to content

CI: full box-lifecycle regression on the bench; sign the runner in to the box access gateway#144

Merged
danielrmerskine merged 4 commits into
mainfrom
de/box-lifecycle-ci
Jul 23, 2026
Merged

CI: full box-lifecycle regression on the bench; sign the runner in to the box access gateway#144
danielrmerskine merged 4 commits into
mainfrom
de/box-lifecycle-ci

Conversation

@danielrmerskine

Copy link
Copy Markdown
Collaborator

Summary

Two changes to bench CI:

  1. update-regression.yml becomes Box Lifecycle: the weekly job previously tested only lager update --force — the path users hit least — and has been aborting at the update confirmation prompt it predates. It now walks the bench box through every lifecycle path users actually run, in an order where each step sets up the next and the box ends the run freshly installed on current main.
  2. Both bench workflows sign in to the box access gateway at job start. Gateway auth landed in v0.32.0; every bench run since has died at "Box requires sign-in" before reaching hardware. The runner now logs in with a dedicated CI account (vars.LAGER_AUTH_URL + LAGER_CI_EMAIL/LAGER_CI_PASSWORD secrets, least-privilege: member role with a direct assignment to the bench box only).

What the lifecycle run proves, step by step

step proves
lager update --check the dry run is read-only and healthy
update --version <previous tag> version pins + the rollback path (target floored at v0.32.0: a current CLI cannot drive a pre-migration box)
plain lager update (N-1 -> main) the common path — a release that breaks upgrading from the previous release goes red here
plain update again + output assert the "already at version" fast path actually engages (incl. its container-liveness gate)
lager update --force the clean-rebuild path (what the old workflow tested)
lager uninstall (scoped) + lager install + smoke a box can be rebuilt from nothing by the current tooling

lager hello gates every phase; a Rigol supply smoke test runs after the update phases and again after the reinstall so failures attribute cleanly.

The deliberately scoped uninstall

  • Never --all: it removes the runner's key from the box's authorized_keys, and CI has no TTY for install's password fallback — that would strand the bench until someone re-keys it by hand.
  • Always --keep-config: /etc/lager holds the saved bench nets and control_plane.json (the box's gateway enrollment); wiping either breaks the rest of the run.
  • install goes by --ip because uninstall --yes removes the local box entry, then the box is re-registered.
  • A failure()-gated recovery step does one best-effort reinstall so the uninstall->install window cannot leave the bench dead for a week; the run itself stays red.

Testing

  • actionlint clean on all three workflow files (only the expected unknown-custom-label notes for lager-bench).
  • Supervised workflow_dispatch of both workflows from this branch before merge — a real box update/reinstall cannot be rehearsed in a container; results will be posted here.

Notes

  • Cadence, concurrency group (serialized against integration-tests.yml on the one physical bench), and the no-pull_request posture (public repo + self-hosted runner) are unchanged.
  • Supersedes the pending one-line --yes fix for the old workflow (same file, superset).

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

update-regression.yml tested only `lager update --force` (the path users
hit least) and predates the update confirmation prompt, so it has been
aborting at the prompt. Rebuild it as a Box Lifecycle workflow covering
the paths users actually run, each verified with `lager hello` and ended
with a hardware smoke test:

- `lager update --check` dry run
- downgrade to the previous release tag (version pin + rollback path,
  floored at v0.32.0 - a current CLI cannot drive a pre-migration box)
- common-path `lager update` (previous release -> main), the invocation
  users run most; a release that breaks upgrade-from-previous goes red
  here
- no-op update, asserting the "already at version" fast path (and its
  container-liveness gate) actually engages
- forced clean rebuild (--force)
- scoped uninstall + from-scratch install. Scope is load-bearing:
  --keep-config preserves the saved nets AND the box's control-plane
  enrollment; --all is deliberately never used because it removes the
  runner's key from authorized_keys and CI has no TTY for install's
  password fallback
- a best-effort recovery reinstall on failure so a red run cannot
  strand the bench for a week

Both bench workflows now also sign in to the box access gateway at job
start (vars.LAGER_AUTH_URL + CI account secrets); box gateway auth
landed in v0.32.0 and bench runs have failed at "requires sign-in"
since. The doubled `lager hello` covers the gateway's one-time
first-contact link recording after a fresh login.
@danielrmerskine

Copy link
Copy Markdown
Collaborator Author

Supervised validation complete: run 30050163152 is fully green end to end — all five update paths (dry run, downgrade to previous tag, common-path update, fast-path assert, forced rebuild), hardware smoke, scoped uninstall, headless from-scratch install, and a final hardware smoke on the freshly installed box.

The six supervised runs before this one each surfaced a real defect, all now fixed or worked around:

  1. install/uninstall SSH-probe uses the default identity — fixed here via the runner ssh-config stanza.
  2. CLI gateway-auth refresh storm against short-TTL tokens — fixed in Gateway auth: adaptive refresh margin, stale-token fallback, replay-safe retry #145.
  3. Headless install fails on interactive sudo with a misleading "Invalid sudoers syntax" error — bench precondition documented; honest pre-check tracked as a follow-up.
  4. install/uninstall removed containers and images they do not own — fixed in Install/uninstall: only remove the containers and images lager owns #146.
  5. install leaks its box auto-lock (single-shot release races the deploy's docker restart; same-job unlock is refused as a different holder) — force-unlock workarounds in this workflow, documented in-file; product fix tracked as a follow-up.

The workflow's recovery step now also fires on cancellation (a cancelled mid-install run is the one observed way to strand the bench).

@danielrmerskine
danielrmerskine merged commit d1cf7d2 into main Jul 23, 2026
6 checks passed
@danielrmerskine
danielrmerskine deleted the de/box-lifecycle-ci branch July 23, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant