ci: enrich scheduled-failure issues with an LLM triage pass

[tonyandrewmeyer]

Today, when a scheduled workflow fails, notify-scheduled-failure.yaml opens an issue whose title and body are a generic one-liner plus a link to the job. A human then has to read the logs, check whether it duplicates an existing issue, and rewrite the title and body before it is useful to anyone.

This adds a second stage that does that first pass automatically: extract a deterministic failure signature from the failing jobs' logs, dedupe against real candidate issues, ask an LLM to draft a descriptive issue or comment, validate its output against a schema, and apply it — falling back to today's plain notice at every layer if anything goes wrong.

Two stages, not one

Stage 1 (notify-scheduled-failure.yaml) keeps its current guarantee: no secrets, no LLM, no network beyond gh, always produces a notification. The only change is a coarse dedupe — it searches open issues for the workflow name and comments on a match instead of opening a duplicate.

Stage 2 (ai-failure-enrich.yaml, new) does the enrichment. It is a separate workflow_run-triggered workflow rather than being spliced into the notifier, so that a broken or unprovisioned stage 2 cannot affect whether a notification happens at all.

Note that workflow_run cannot target the notifier directly — a reusable workflow invoked via workflow_call does not get an independent run to subscribe to. So the enricher lists the seven scheduled workflows that call the notifier, by their exact name:, and gates on conclusion == 'failure' && event == 'schedule', mirroring the callers' own if: failure() && github.event_name == 'schedule'.

Dedupe ladder

1. Marker match — if a :sig= marker for this exact run id already exists anywhere in the repo's issues, this is a re-run of the same failing jobs. Comment "re-run attempt still failing" and skip signature extraction and the LLM call entirely. On a ten-run calibration corpus this caught 2 of the 4 real duplicate pairs — the single highest-value rung, and the reason both real July duplicates existed at all was re-runs rather than recurring signatures.
2. No API key → plain fallback body, still marker-stamped.
3. Candidate search — open issues plus recently-closed ones (≤14 days, capped at 3, and explicitly labelled "(closed …)" in the block the model sees, with an instruction never to auto-treat a closed issue as a strong match) feed the LLM's comment-vs-new decision.
4. Schema-invalid output → plain fallback body, with the validation errors written to the job's step summary.
5. Valid new envelope on a fresh placeholder → gh issue edit the placeholder in place, rather than opening a second issue.
6. Valid new envelope where stage 1 had commented on an older issue → the coarse match was wrong: open a genuinely new issue and leave a pointer comment on the older one.

The marker format is <!-- ai-failure-notifications:run=<id>:origin=new|comment --> from stage 1 and <!-- ai-failure-notifications:run=<id>:sig=<hash> --> from stage 2. Rung 1 keys specifically off :sig=, so a same-run re-fire arriving before enrichment finishes is not mistaken for "already handled".

The coarse search matches on workflow name alone rather than the full "Scheduled workflow 'X' failed" sentence, because enrichment rewrites the title and would otherwise break its own future matching. The applier always appends a Workflow: <name> footer so the match survives.

Failure handling

A notification is never lost, and a lost enrichment does not cost a duplicate issue:

1. Inside the script — no API key, OpenRouter error, or schema-invalid output falls back to the plain body through the same marker-aware applier. gh search failures degrade to "no marker" / "no candidates" rather than raising.
2. Inside the workflow — if enrich crashes outright, the plain-fallback job fires. It is marker-aware for the same reason everything else here is: by the time it can run, stage 1 has already notified, because workflow_run: completed only fires once the caller's run — open-issue job included — has finished. Creating an issue unconditionally therefore meant two issues for one failure whenever enrich died. It now looks for ai-failure-notifications:run=<id>: and comments on that issue instead, creating one only when the marker is genuinely absent, which is the one case that means stage 1 itself failed. The trailing colon matches the notifier's :origin= and the enricher's :sig= alike, so it also covers enrich having applied its result and then died before setting handled. It lists recently updated issues rather than searching, for the read-your-writes reason described at rung 1.
3. Stage 1 never depends on stage 2 at all.

Please look at the zizmor suppression

This adds the first # zizmor: ignore[...] in the repository. zizmor raises dangerous-triggers (high, confidence medium) on the enricher's workflow_run trigger.

The two exploitable workflow_run patterns the audit exists to catch are downloading an artifact from the triggering — potentially fork-controlled — run and trusting it, and checking out github.event.workflow_run.head_sha to execute it with this workflow's secrets and write token. Neither is present:

• no artifact is downloaded;
• actions/checkout takes no ref, so it checks out GITHUB_SHA, which for workflow_run is the default branch tip and never the triggering run's ref, with persist-credentials: false;
• every github.event.workflow_run.* value reaches the script through env:, never interpolated into a run: block, so there is no template injection surface;
• both jobs gate on github.event.workflow_run.event == 'schedule'. A fork pull request produces event == 'pull_request', so a fork cannot trigger this at all — every run that reaches it originates from schedule: on the default branch.

workflow_run is also not substitutable here, for the reusable-workflow reason described above: the alternative is splicing an enrichment call into all seven callers, which is the coupling the two-stage design exists to avoid.

If you would rather not take the suppression precedent in a feature PR, the alternative is that seven-caller restructure, and I am happy to do it instead.

Testing

The read-only gh layer has been exercised for real against this repository, using run 29847889218 (the 2026-07-21 "Example Charm Integration Tests" scheduled failure, which opened #2658). On that run the extractor independently produced traceback_top_error: "KeyError: 'loki/0'" along with the three failing tests and their timeout messages, which matches the hand-written diagnosis left on #2658. That is the first check of the extractor against a log it was not calibrated on.

The write calls have now been exercised too, in my fork, against a deliberately-failing stand-in workflow with a real OpenRouter key.

Before this does anything visible

• The ai-failure-triage environment and its OPENROUTER_API_KEY need provisioning. Until then every run takes the "no API key" path — which is still better than today's baseline, since the coarse dedupe now works, and makes this safe to merge behind.
• OPENROUTER_MODEL repo variable — the script defaults to deepseek/deepseek-chat if unset; worth pinning explicitly once the model choice is confirmed against live traffic.

[tonyandrewmeyer]

@benhoyt adding you as a reviewer specifically around the Zizmor/workflow question (feel free to review anything else as well, of course).