Skip to content

fix(auth): enforce the last-administrator invariant inside the write transaction - #9479

Closed
lstein wants to merge 2 commits into
invoke-ai:mainfrom
lstein:fix/last-admin-invariant-in-service
Closed

fix(auth): enforce the last-administrator invariant inside the write transaction#9479
lstein wants to merge 2 commits into
invoke-ai:mainfrom
lstein:fix/last-admin-invariant-in-service

Conversation

@lstein

@lstein lstein commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Kind: fix (backend, auth)

Removing the last active administrator is irreversible from inside the app. Authorization is derived from the database on every request, so the caller loses admin access immediately and no authenticated path back exists. It also drops has_admin() to zero, which makes GET /auth/status report setup_required: true and re-opens the unauthenticated POST /auth/setup to whoever reaches it first.

That invariant was enforced in exactly one place — the delete_user route — and enforced there by reading count_admins() in its own transaction and then writing in another. Three gaps followed:

  1. update_user had no guard at all. PATCH /api/v1/auth/users/{id} with {"is_admin": false} or {"is_active": false} against the sole administrator succeeded, single-threaded, no race required. count_admins() had one call site in the whole codebase.
  2. The check and the write were separate transactions — a TOCTOU. Two callers each observe two administrators and each remove one.
  3. invoke-usermod / invoke-userdel construct UserService directly and never reach the route guard, so the CLIs could take the instance to zero on their own.

The fix

The check moves into UserService.update() and UserService.delete(), evaluated on the cursor of the transaction that performs the write. Those transactions now open with BEGIN IMMEDIATE so the count is read under the write lock — without it the SELECT runs in autocommit and a second process can still interleave between the read and the write. In-process callers are additionally serialized by the database's shared RLock.

The guard keys on the requested values rather than on the target being an administrator, so these stay allowed:

  • renaming or changing the password of the last administrator;
  • demoting one of two administrators;
  • removing an administrator who is already inactive — they are not counted by count_admins(), so removing them cannot reach zero.

LastAdministratorError subclasses ValueError, which the routes (except ValueError -> 400) and both CLIs already handle, so no call site needed changing. The existing route-level check in delete_user stays as the friendly-message path; the service is the backstop.

Why now

Found while reviewing #9436, which converts 167 route handlers from async def to def. An async def handler with no await in its body cannot yield, so its check-then-act ran atomically against other requests; dispatched to the threadpool it does not. That turns gap 2 from a cross-process race into one reachable from two concurrent HTTP requests:

DELETE /api/v1/auth/users/a1   ->  204
DELETE /api/v1/auth/users/a2   ->  204     # two admins existed
admins remaining: 0

This PR is independent of #9436 — the invariant is wrong on main today, gap 1 needs no concurrency at all, and the fix is confined to the user service.

Relationship to #9360

#9360 adds a route-level count_admins() guard to update_user, which closes gap 1 at the API boundary. It does not move either guard into the write transaction, so gaps 2 and 3 survive it. The two changes are complementary and touch different layers: #9360 keeps the friendly 400 in the route, this PR makes the invariant hold underneath it. Expect a small textual conflict in users_default.py only if both land; neither depends on the other.

QA Instructions

Automated. 17 new tests, all verified to fail before the fix and pass after — 9 of the 17 fail without the guard, including every concurrency case:

  • tests/app/services/users/test_last_admin_invariant.py — the guard itself. Rejects delete / demote / deactivate of the last admin; allows rename, password change, demoting one of two, and removing an already-inactive admin. Three tests spawn two threads through a threading.Barrier and assert exactly one succeeds: concurrent deletes, concurrent demotions, and one of each.
  • tests/app/routers/test_last_admin_routes.py — the HTTP contract: 400 rather than a 500 escaping from the service, and the last admin's record is not blanket-locked.

Full backend run on this branch: 2197 passed, 8 skipped, 6 xfailed, 0 failures. ruff check / ruff format --check clean.

Manual. In multiuser mode with a single administrator:

  1. PATCH /api/v1/auth/users/{admin_id} with {"is_admin": false} → 400 Cannot remove the last administrator (before: 200, and the instance falls back to unauthenticated setup).
  2. Same with {"is_active": false} → 400.
  3. {"display_name": "..."} → 200, still works.
  4. invoke-usermod --no-admin <email> on the sole administrator → refused with the same message instead of succeeding.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration — n/a, backend only
  • Documentation added / updated (if applicable) — n/a, no user-facing surface change beyond the new 400
  • Updated What's New copy (if doing a release after this PR)

…transaction

Removing the last active administrator is irreversible from inside the app: authorization
is derived from the database on every request, so no authenticated path back exists. It
also drops `has_admin()` to zero, which makes `GET /auth/status` report `setup_required:
true` and re-opens the *unauthenticated* `POST /auth/setup` to any caller.

The invariant was enforced in one place only — the `delete_user` route — and enforced
there by reading `count_admins()` in its own transaction before writing in another. That
left three gaps:

- `update_user` had no guard at all, so `PATCH /auth/users/{id}` with `{"is_admin": false}`
  or `{"is_active": false}` against the sole administrator succeeded, single-threaded.
- The read and the write were separate transactions, so two callers could each observe two
  administrators and each remove one. Route handlers now run in a threadpool, which makes
  that reachable from two concurrent requests rather than only across processes.
- `invoke-usermod` / `invoke-userdel` construct `UserService` directly and never reach the
  route guard, so the CLI could take the instance to zero on its own.

Moves the check into `UserService.update()` / `UserService.delete()`, evaluated on the
cursor of the transaction that performs the write. Those transactions now open with
`BEGIN IMMEDIATE` so the count is read under the write lock: without it the SELECT runs in
autocommit and a second process can still interleave. In-process callers are additionally
serialized by the database's shared RLock.

The guard keys on the *requested* values, so renaming or changing the password of the last
administrator stays allowed, as does removing an administrator who is already inactive —
they are not counted, so removing them cannot reach zero.

`LastAdministratorError` subclasses `ValueError`, which both the routes and the CLIs
already map to a friendly 400 / error message, so no call site needed changing. The
existing route-level check in `delete_user` stays as the friendly-message path; the
service is the backstop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added python PRs that change python files services PRs that change app services python-tests PRs that change python tests labels Aug 8, 2026
@lstein lstein added the 6.14.1 label Aug 8, 2026
@lstein lstein moved this to 6.14.1: Bug fixes to 6.14.0 in Invoke - Community Roadmap Aug 8, 2026
@lstein

lstein commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #9360 rather than landed separately.

Both PRs were reworking the same invariant from opposite ends — #9360 added the
update_user route guard and #9479 moved the check into the write transaction — so
keeping them apart meant two reviews of one change and a guaranteed conflict in
users_default.py. The commit here is cherry-picked onto #9360 unchanged
(e9317f6583), with follow-ups on top.

Merging them surfaced things neither PR could see alone:

  • The invariant was enforced with the wrong predicate, in both layers. The system
    row is active but has an empty password hash, so it can never authenticate — yet
    count_admins() counts it. Promoting it took the count 1 → 2, which was enough for
    both guards to then allow the last real administrator to be demoted, leaving no
    usable administration and /auth/setup still closed. Setting a password on that row
    was the same hole from the other end. Adding a second enforcement layer had duplicated
    the wrong predicate rather than fixing it. The system-user protection now lives in the
    service beside the last-admin guard, so invoke-usermod/invoke-userdel are covered
    too, and a migration demotes the row on databases where it was already promoted.
  • This PR's tests could not run on fix(auth): revoke privileges immediately on role change, deactivation, or deletion #9360's schema. The fixture hand-builds the
    users table and fix(auth): revoke privileges immediately on role change, deactivation, or deletion #9360 adds a token_epoch column that every UserService query
    selects, so all 12 tests failed with no such column.
  • Three further defects in fix(auth): revoke privileges immediately on role change, deactivation, or deletion #9360 found by the same review: a KeyError in the socket
    re-authorization loop that silently abandoned the remaining sockets on stale
    privileges, a cancellation that acted on a stale snapshot, and update_user returning
    400 instead of 404 for an unknown user id.

Everything from here is preserved in #9360, including the concurrency tests. Continuing
there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.1 python PRs that change python files python-tests PRs that change python tests services PRs that change app services

Projects

Status: 6.14.1: Bug fixes to 6.14.0

Development

Successfully merging this pull request may close these issues.

2 participants