Dependabot auto-merge: AI risk gate for every major bump + CI gate on every merge

[bbkrr]

Problem

The AI risk gate (RealRate-Private#2229) is live and decides eligibility for major github-actions bumps from the changelog. But the actual merge, for both patch/minor and AI-approved majors, used gh pr merge --auto — which only waits for required status checks.

I verified that across our repos there are effectively no required status checks configured (RealRate-Private's develop has none — no ruleset check, no classic branch protection). So today a bump auto-merges even when CI is red. The tests run; they just don't block anything. The AI verdict is effectively the only gate.

Why not just "make the tests a required check"?

Required checks are matched by exact name, per repo. This workflow is injected org-wide across repos whose test jobs are named differently (or that have none). A single org rule can't name them all, and requiring a check a repo doesn't emit would deadlock every PR in that repo. So required-checks don't scale to the org here.

Approach — read the PR's own checks in the shared workflow

New review-tooling/gated_merge.sh; every auto-merge path (patch/minor and AI-approved majors) now routes through it instead of gh pr merge --auto. It:

1. Waits for the PR's checks — every check except this workflow's own run (self-excluded by run id inside the check link) — to reach a terminal state.
2. Merges only if all passed. Any fail/cancel bucket, or a timeout, routes the PR to needs-manual-review instead.

Because it reads the PR's actual check conclusions, it works uniformly regardless of what each repo names its checks — org-wide "don't merge over red CI" with zero per-repo config. This is the "readiness" half of the #2229 design ("AI decides eligibility; CI decides readiness"); the AI still does the changelog judgment, this just refuses to merge over red CI.

Safety properties

• Fail-closed. The job is itself a required workflow, so the script always exits 0 and routes to a human on any failure/timeout rather than merging.
• Kill switch. CI_GATE_ENABLED: "false" reverts to native --auto org-wide via one env flip, no revert needed.
• No-CI repos unchanged. After a grace period with no checks registered, it merges on the existing signal (patch/minor policy or AI verdict) — same as today.
• Per-PR concurrency cancels superseded runs so two runs never race to merge the same PR.

Resulting behavior on a Dependabot PR

Case	Outcome
patch/minor, CI green	auto-merge
patch/minor, CI red	held for human (was: merged)
major github-actions, AI auto, CI green	auto-merge
major github-actions, AI auto, CI red	held for human (was: merged)
AI human/uncertain/error, or non-allowlisted major	held for human (unchanged)

Validation

• shellcheck + actionlint clean.
• All decision branches (green → merge, red → hold with failing-check names, pending → keep waiting, no-checks → grace) dry-run-verified against live gh pr checks JSON.
• Not yet exercised end-to-end — that needs a real freshly-opened Dependabot bump (the path only runs on opened). Suggest merging with CI_GATE_ENABLED first observed on the next real bump, or temporarily pointing a test bump at it.

Honest caveats to review

• Appearance grace race: if a repo's checks take longer than GATE_APPEAR_SECONDS (180s) to register, the script could treat it as a no-CI repo and merge. Dependabot checks normally register in seconds; tunable, and the robust-but-non-scaling alternative is native required checks per repo.
• Rebase during the CI wait: concurrency: cancel-in-progress cancels the in-flight run; for an AI-gated major the verdict is computed only on opened, so a mid-wait rebase drops to manual review (safe, but no auto-merge on that PR).

Refs: RealRate-Private#2244 (this CI gate), RealRate-Private#2229 (AI gate). Ships behind CI_GATE_ENABLED (default on).

🤖 Generated with Claude Code

[bbkrr]

Update: widened the AI gate to all repos (2nd commit 5da5011)

Per follow-up scope decision, the AI risk verdict now runs for major github-actions bumps in every repo the shared workflow is injected into, not just the RealRate-Private pilot. Removed AIGATE_ALLOWLIST entirely.

Unchanged: patch/minor (auto, CI-gated), and major pip/docker (comment + needs-manual-review for a human). The generic major-bump comment/label now fire only for non-github-actions majors.

Decision map now:

Bump	Repo	Path
patch / minor	any	auto-approve → CI-gated merge
major, github-actions	any	AI verdict → auto merges (CI-gated), else human
major, pip/docker	any	comment + needs-manual-review

Still fail-closed (AI error → human); enforce vs observe still controlled by AIGATE_DRY_RUN (currently false = enforcing). actionlint clean.

⚠️ Note for reviewers: this flips the AI from a single-repo pilot to org-wide enforcing in one step. If you'd rather observe verdicts across repos first, set AIGATE_DRY_RUN: "true" before merge and flip it after watching a few real bumps.

[bbkrr]

Update: AI gate now covers every major bump, all ecosystems (commit f6ba898)

Per follow-up: the AI risk verdict now runs for all major bumps — pip, docker, and github-actions — in every repo, not just github-actions.

• Dropped the package-ecosystem == 'github_actions' condition from the AI verdict / auto-merge / route-to-human steps.
• Removed the now-redundant generic major-bump comment+label steps (every major is handled by the AI block).
• Generalized the classifier prompt from Actions-specific ("pinned Action", "inputs/outputs", "Node version") to ecosystem-aware — it now judges a library (pip), a base image (docker), or a CI action, and is told the ecosystem via a new --ecosystem arg from fetch-metadata. Same strictness: any breaking public-API / input / default / platform-support / security change, or any uncertainty → human.

Decision map now:

Bump	Repo	Path
patch / minor	any	auto-approve → CI-gated merge
major (pip / docker / github-actions)	any	AI verdict → auto merges (CI-gated), else human

Still fail-closed (AI error → human), still CI-gated by gated_merge.sh, enforce/observe via AIGATE_DRY_RUN. actionlint clean; script compiles; fail-closed fallback verified offline.

⚠️ Reviewer note — this is the biggest scope jump. pip major bumps are qualitatively riskier than github-actions bumps: they're real code-API breaks, and (per RealRate-Private#2244's coverage analysis) our test suite mocks the IO-tier libs (requests/yfinance/sec_api/google) and doesn't exercise scipy — so for those, green CI is weaker evidence and the AI verdict is carrying more weight. Strongly suggest shipping AIGATE_DRY_RUN: "true" first to observe pip/docker verdicts before enforcing.

[bbkrr]

Correction (commit 2614081): this repo's dependabot.yml declares package-ecosystem: uv for Python (not pip), plus github-actions and docker. Updated the wording where I'd written "pip". No behavior change — the AI gate is ecosystem-agnostic (ecosystem is a hint, not a gate), so it covers uv major bumps regardless of the exact string fetch-metadata reports. Prior comments on this PR that say "pip" should read "uv".

[bbkrr]

Set to observe mode (AIGATE_DRY_RUN: "true", commit f6c7cc6) per the recommendation. On merge, every major bump (uv/docker/github-actions, any repo) gets its AI verdict posted as a comment and is routed to a human — nothing auto-merges on the AI verdict yet. patch/minor auto-merge and the CI gate are unaffected. Flip the flag to "false" to enforce once the verdicts look right on real bumps.

[bbkrr]

Final aligned behavior (commit f91a898, ENFORCING)

Confirmed the target with the team. This is what the workflow now does on a Dependabot PR:

Bump	Flow	Outcome
patch / minor (any ecosystem)	approve → CI	auto-merge if CI green
major — libraries (uv)	AI risk check → CI	auto-merge if AI low-risk AND CI green, else human
major — github-actions	AI risk check → CI	auto-merge if AI low-risk AND CI green, else human
major — docker	AI risk check → CI	auto-merge if AI low-risk AND CI green, else human

• AI-flagged risk, or any failed check, or an AI error → routed to a human (needs-manual-review), never merged.
• Applies org-wide (every repo the shared workflow is injected into).
• AIGATE_DRY_RUN: "false" (enforcing) · CI_GATE_ENABLED: "true" (CI gate on).

Both flags remain as fast kill switches if anything misbehaves. Ready for review.

[bbkrr]

AI tooling now uses uv (not pip/requirements.txt) — commit 8a6826d

Aligned with the org's uv standard: review_dependabot_major.py declares its single dep inline via PEP 723 and runs with uv run --script (behind a pinned astral-sh/setup-uv step). Deleted requirements.txt; dropped the pip install.

--script forces script mode so uv ignores the caller repo's pyproject.toml/project env — verified it runs even from a dir with an unresolvable pyproject. Validated e2e: uv provisions openai in an isolated env and the fail-closed path still returns verdict=human with no token. (Reminder: openai is just the client lib pointed at GitHub Models; no OpenAI key/billing.)