Use a PAT for Models inference so the major-bump AI gate actually runs

[bbkrr]

Refs #14.

Why

The AI half of the major-bump gate has never produced a verdict in production. Both real majors so far (EBA#101 numpy, EBA#105 setup-uv) were routed to a human because the Models call 403s, not because the AI judged them risky.

What I measured

Probed from inside Actions, in an org repo, on the real GITHUB_TOKEN (temp branch, since deleted):

call	result
catalogue read /catalog/models	200 (returns 37 models incl. openai/gpt-4.1)
inference — 4 × low tier (gpt-4.1-mini, gpt-4o-mini, phi-4, mistral-small)	403, empty body
inference — 2 × high tier (gpt-4.1, gpt-4o)	403, empty body
inference — 2 × custom tier (gpt-5-mini, o4-mini)	403, empty body
same inference call, user token	200

The job log confirms Models: read is granted. So: auth works and the catalogue reads, but every inference is refused regardless of model or tier. It is not a workflow permission, not pull_request_target, not Dependabot's restricted token, and not the model id. The org simply has no Models inference entitlement, while a user account does.

Ruled out along the way: the org's Copilot model list (GPT-5.5, Claude Opus 4.8) is a different product with a different catalogue — openai/gpt-5.5 returns 404 unknown_model, and the Models catalogue has no Anthropic publisher at all. Enabling Copilot models cannot help here.

Changes

• Prefer MODEL_TOKEN over GITHUB_TOKEN. The old order checked GITHUB_TOKEN first, and that is always set in Actions — so a supplied PAT would have been silently ignored and kept 403ing. This one-line ordering bug would have made the whole PAT approach look broken.
• Pass MODEL_TOKEN from a new MODELS_PAT secret.
• Explain the 403. GitHub returns an empty body, so the bare PermissionDeniedError told maintainers nothing and sent us hunting through workflow permissions. It now names the likely cause and links AI risk gate: GitHub Models call returns 403 — major-bump AI verdict never runs #14.
• Fix the docstring, which claimed the built-in token authenticates inference.

Safety

Unchanged, and fails safe. If MODELS_PAT is absent the secret renders as "", the script falls back to GITHUB_TOKEN, gets a 403, and emits verdict=human — exactly today's behaviour. Any error still routes to a human, and gated_merge.sh still requires the PR's own checks to pass, so no bump can merge on an errored verdict. This PR can land before the secret exists and change nothing until it does.

Validation

• New logic exercised across all paths: precedence, fallback, empty-secret (the undefined-secret case), no-token → verdict=human, 403 → actionable message (by status_code and SDK class name), non-403 errors unchanged.
• Repo gate green locally: test_gated_merge.sh ALL PASS, shellcheck clean, script compiles, workflow YAML parses.

⚠️ Needs an admin before it does anything

1. Create a fine-grained PAT with Models: read only — no repo/contents/org scopes. Least privilege: it does inference and nothing else. Ideally on a machine account, not a person, so it survives staff changes.
2. Add it as an org-level Actions secret named MODELS_PAT, visibility all repositories (the gate runs in each target repo, so a secret on .github alone won't reach it).

Caveats worth knowing: inference is attributed to that account's allowance and is rate-limited (fine for major bumps, which are rare), and the token needs rotating like any credential. The cleaner long-term fix remains org Models entitlement/billing, which needs no credential at all — this unblocks us meanwhile.

🤖 Generated with Claude Code

[bbkrr]

Security review of this diff

Putting a credential into a pull_request_target workflow deserves scrutiny, since that trigger plus untrusted code is the classic exfiltration pattern. It does not apply here, and I checked rather than assumed:

• No untrusted code ever runs. The only checkout is realrate/.github itself, pinned to a full SHA (93cb6ef, L82-85). The PR's head is never checked out, built, or executed.
• Only Dependabot can reach the secret. The job is gated on github.event.pull_request.user.login == 'dependabot[bot]' (L69). PR authorship is set by GitHub, so it cannot be spoofed by opening a PR.
• Least privilege by design. The PAT should carry Models: read and nothing else, so the worst case if it leaks is someone burning that account's inference quota — no repo, code, or org access.
• No injection surface added. The PR body reaches the script via --body-file (written by gh pr view ... > pr_body.txt), never interpolated into a shell command.
• Fails safe. Absent secret → "" → falls back to GITHUB_TOKEN → 403 → verdict=human. Errors cannot produce an auto-merge, and gated_merge.sh still demands green CI.

Pre-existing residual risk, not introduced here, worth stating out loud: the changelog we feed the model is untrusted upstream text, so a hostile release note could attempt prompt injection toward verdict=auto. The CI gate is the backstop (a bump that breaks CI never merges regardless of verdict), and the prompt already labels the body untrusted. Worth a follow-up to harden if we ever drop the CI gate.

[bbkrr]

✅ End-to-end test on the two real major bumps that never got a verdict

Ran this branch's script, unmodified, with a real PAT in MODEL_TOKEN, against the real GitHub Models API (openai/gpt-4.1), fed the real PR titles + changelogs from the two majors that fell through to manual review.

Case 1 — EBA#105, astral-sh/setup-uv 6 → 7 (github_actions)

{"verdict": "auto", "risk": "low",
 "reason": "There is no explicit indication of breaking changes; updates are maintenance, documentation improvements, dependency bumps, support for a new architecture, and internal changes; no public API or input/output changes described."}

Matches reality: a human reviewed and merged it, CI green. This one would have merged itself, still gated on green CI.

Case 2 — EBA#101, numpy 1.26.4 → 2.5.0 (pip)

{"verdict": "human", "risk": "high",
 "reason": "NumPy 2.5.0 drops support for Python 3.11, removes distutils, and expires multiple deprecations which may break usage or compatibility for typical consumers, plus renames/removes some APIs; these are breaking changes."}

It named the actual root cause, unprompted. #101's CI really did fail because numpy 2.5 needs Python >=3.12 while tests.yml pinned 3.11 — the exact thing we spent a session diagnosing by hand on 2026-07-15, and precisely what #102 then fixed. The model surfaced it straight from the changelog, and correctly refused to auto-merge.

So the gate discriminates: it waves through the safe major and stops the breaking one for the right reason. Not a rubber stamp.

What this proves: MODEL_TOKEN precedence works, gpt-4.1 is reachable and returns structured JSON, and the verdicts are sound on real data.

What it does not prove: the same call from inside the Actions runner with MODELS_PAT set. That is the one step needing the org secret; the credential is user-attributed either way, so I expect it to behave identically, but I'll confirm on the first real major bump rather than assume.

[bbkrr]

Added: models-selftest — verify the gate on demand, instead of waiting for a major bump

Testing this by waiting for a real major bump to wander in (and reading its comment on a live PR) is a bad feedback loop. This adds a self-test you can run whenever you like:

gh workflow run models-selftest.yaml --repo realrate/.github
# or: Actions -> models-selftest -> Run workflow

It probes inference with the built-in token and the PAT side by side, then runs the real script over two fixtures with known answers, and renders a summary table.

Three design decisions worth reviewing

1. It fails LOUD — the opposite of the gate. The gate fails safe (any error → verdict=human), which is correct for merging but is exactly why a dead AI went unnoticed for weeks. A diagnostic must go red instead.

2. It rejects fallback verdicts, not just malformed ones. This is the subtle part. The script emits verdict=human on any error — which matches the breaking fixture's expected answer. So a total outage would sail through a naive test. Verified by running with a deliberately broken token:

EXIT=1
::error::breaking: verdict came from the error fallback, not the model: AI verdict unavailable (AuthenticationError); manual review.

Detection keys off the fallback's signature (risk=unknown and reason ending manual review.), so a genuine model verdict that happens to mention manual review won't trip it.

3. Judgement drift warns; only a missing/fallback verdict fails. The model is non-deterministic, so a differing call is reported, not fatal.

Verified

Real API, real script, both fixtures:

fixture	expected	verdict	risk
breaking (numpy-style: drops Py3.11, removes distutils, ABI change)	human	✅ human	high
safe (action bump, no input/output changes)	auto	✅ auto	low

Broken token → exit 1 and red, as shown above. And in this PR's actual CI run, with no secret configured, it did the right thing: detected the absent secret, emitted a notice, skipped the three live steps, and stayed green — so it never blocks PRs before the secret exists.

🐛 Bug this testing exposed

MODEL_ID defaulted to openai/gpt-5, which is reasoning-tier and rejects this script's max_tokens with a BadRequestError. The documented default never worked — only the workflow's explicit override did, which is why it went unnoticed. Now defaults to openai/gpt-4.1, verified working with MODEL_ID unset.

After the secret lands

Run the workflow. Green means majors will start getting real verdicts. If it also reports the built-in token returning 200, the org entitlement got fixed and MODELS_PAT can simply be deleted.

[bbkrr]

✅ Resolved: a personal-account token does inference from Actions (proven)

Self-test run from inside Actions, on realrate/.github:

credential	inference
built-in GITHUB_TOKEN	403
personal classic PAT (via MODELS_PAT_TEST)	200

Real script over both fixtures: breaking bump → human/high, safe bump → auto/low. All steps ran (none skipped), job green.

Conclusion: the Actions runner does not force org-attribution. Inference with a personal PAT is fine. The earlier MODELS_PAT 403 was the token, not the context — it was created org-owned/fine-grained, so it inherited the org's missing inference entitlement. A plain classic token owned by a personal account (no Models scope needed) is the working shape.

Last step for production: put that same working token into the real MODELS_PAT at org level so the gate reaches every repo:

gh secret set MODELS_PAT --org realrate --visibility all   # needs org-owner

Then delete the temporary MODELS_PAT_TEST and the gate is live. (The still-cleaner long-term fix remains org Models billing, which needs no token at all — but this unblocks us now.)