Skip to content

feat: Add check_run webhook support for CI failure auto-remediation#1558

Open
knechtionscoding wants to merge 2 commits into
kelos-dev:mainfrom
datagravity-ai:kelos-946-check-run-webhook
Open

feat: Add check_run webhook support for CI failure auto-remediation#1558
knechtionscoding wants to merge 2 commits into
kelos-dev:mainfrom
datagravity-ai:kelos-946-check-run-webhook

Conversation

@knechtionscoding

@knechtionscoding knechtionscoding commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds first-class support for CI/CD failure auto-remediation by teaching the
GitHub webhook path to understand check_run events. Previously check_run
events fell through to the generic default handler (only sender + action
extracted), and GitHubWebhookFilter had no way to filter by conclusion or
check name — so a CI auto-fix spawner would trigger on every completed check run
(pass or fail) and could not distinguish lint from test from build failures.

This PR makes the framework changes needed to build a spawner that watches for a
specific failing check and dispatches an agent to fix it, all additive and
backward-compatible:

  • API (api/v1alpha2): two new optional fields on GitHubWebhookFilter:
    • conclusion — filters check_run events by conclusion (kubebuilder enum:
      success, failure, cancelled, timed_out, action_required,
      neutral, skipped, stale).
    • checkName — filters check_run events by check run name (exact match or
      glob, e.g. lint, build-*).
  • Parsing (ParseGitHubWebhook): a *github.CheckRunEvent case that
    populates repository info, Title (check name), HeadSHA, URL, and the
    associated PR Branch/Number.
  • Matching (matchesFilterWithoutFilePatterns): conclusion (exact) and
    checkName (glob) matching for check_run events.
  • Templates (ExtractGitHubWorkItem): new CI-specific variables
    {{.CheckName}}, {{.Conclusion}}, {{.CheckRunURL}}, {{.HeadSHA}},
    {{.CheckApp}}.
  • Reporting (webhookSourceKind): maps check_run to pull-request when
    the check is linked to a PR, so status reporting attaches to the PR.

go-github/v66 already provides github.CheckRunEvent, so no new dependency is
introduced.

Which issue(s) this PR is related to:

Fixes #946

Special notes for your reviewer:

Does this PR introduce a user-facing change?

TaskSpawner/SessionSpawner `githubWebhook` filters now support `check_run` events: added `conclusion` and `checkName` filter fields and new template variables (`CheckName`, `Conclusion`, `CheckRunURL`, `HeadSHA`, `CheckApp`) for building CI-failure auto-remediation pipelines.

Summary by cubic

Adds first-class check_run webhook support to enable CI failure auto-remediation. Spawners can now target specific failing checks by conclusion and name with better context for templates and reporting.

  • New Features
    • api/v1alpha2: add optional conclusion and checkName to GitHubWebhookFilter for check_run events (exact and glob match).
    • Parse check_run payloads to extract repo, check name (Title), HeadSHA, URL, and linked PR Branch/Number.
    • Match check_run events by conclusion (exact) and checkName (glob).
    • Expose template vars: CheckName, Conclusion, CheckRunURL, HeadSHA, CheckApp.
    • Map check_run to pull-request reporting when linked; regenerate CRDs; add docs/tests and an example (examples/17-taskspawner-ci-remediation) that names tasks by check run ID, removes excludeAuthors, and guards PR-less runs with index/if.

Written for commit d0f57bf. Summary will update on new commits.

Review in cubic

Extend the GitHub webhook path to handle check_run events so a TaskSpawner
can watch for a specific failing CI check and dispatch an agent to fix it.

- Add conclusion and checkName filter fields to GitHubWebhookFilter (v1alpha2)
- Parse check_run events (name, head SHA, URL, associated PR branch/number)
- Match check_run events by conclusion (exact) and checkName (glob)
- Expose CheckName, Conclusion, CheckRunURL, HeadSHA, CheckApp template vars
- Map check_run to the pull-request reporting source when linked to a PR
- Regenerate CRDs; add filter, parse, extract, and conversion tests
- Document the new fields and add a CI-remediation example

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature needs-triage needs-priority needs-actor release-note labels Jul 24, 2026
@knechtionscoding
knechtionscoding marked this pull request as ready for review July 24, 2026 09:00

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread examples/17-taskspawner-ci-remediation/taskspawner.yaml
@knechtionscoding

Copy link
Copy Markdown
Contributor Author

@gjkim42 relatively small one, but ready for review

@gjkim42

gjkim42 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

/kelos api-review

@kelos-bot

kelos-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

🤖 Kelos Task Status

Task kelos-api-reviewer-issue-comment-2427726db04e has succeeded. ✅

@kelos-bot

kelos-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

🤖 Kelos API Reviewer Agent @gjkim42

API Design Review

Verdict: REQUEST CHANGES
Scope: Adds v1alpha2 conclusion and checkName filters for GitHub check_run webhooks, plus check-run template and reporting context.

Findings

Compatibility

  • [P0] internal/conversion/taskspawner_test.go:265 and internal/conversion/taskspawner.go:30 — The new test codifies dropping Conclusion and CheckName during down-conversion even though TaskSpawner v1alpha1 remains served. A client that reads and updates a v1alpha2 object through v1alpha1 will silently erase these filters, broadening a filter such as “failed lint check” to every completed check_run. Preserve the opaque v1alpha2-only filter data across the v1alpha1 round trip, following the existing nameTemplate preservation pattern, and test the complete v1alpha2 → v1alpha1 → v1alpha2 round trip.

Validation

  • [P1] api/v1alpha2/taskspawner_types.go:482 — The conclusion enum omits startup_failure, which GitHub defines as a possible check suite or check run conclusion in CheckConclusionState. Because the CRD rejects values outside the enum, users cannot target these legitimate failures. Add startup_failure to the marker, godoc, reference/example documentation, generated CRDs, and matching tests.

Documentation

  • [P2] api/v1alpha2/taskspawner_types.go:916 — The Branch, PromptTemplate, and NameTemplate godoc blocks explicitly enumerate GitHub webhook variables but omit CheckName, Conclusion, CheckRunURL, HeadSHA, and CheckApp, even though the example uses them in all three template surfaces. Update the three field comments (lines 916, 927, and 957) and regenerate the CRDs so the schema documentation matches the implemented contract.

Suggestions

  • Define a named string type and documented constants for the conclusion enum instead of exposing it as an untyped string; this follows Kubernetes enum conventions and gives Go clients compile-time vocabulary for the supported values.

/kelos needs-input

@gjkim42

gjkim42 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Since it's alpha, we don't want to consider the droped fields by roundtrip.

@gjkim42

gjkim42 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

/kelos review

@kelos-bot

kelos-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

🤖 Kelos Task Status

Task kelos-reviewer-issue-comment-42e719ec1975 has succeeded. ✅

@kelos-bot

kelos-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

🤖 Kelos Reviewer Agent @gjkim42

Review Summary

Verdict: REQUEST CHANGES
Overall correctness: patch is incorrect
Scope: Adds check_run webhook parsing, filtering, template variables, PR reporting context, generated CRDs, documentation, and a CI-remediation example.

Findings Overview

Priority Count File:Line Summary
P0 0 none
P1 2 examples/17-taskspawner-ci-remediation/taskspawner.yaml:48 PR-only templates fail on valid check runs without linked PR metadata
P1 2 examples/17-taskspawner-ci-remediation/taskspawner.yaml:16 The example excludes the GitHub Actions sender modeled by this PR
P2 2 examples/17-taskspawner-ci-remediation/taskspawner.yaml:52 The Task name suppresses later check runs for the same PR and check
P2 2 internal/webhook/handler.go:886 The new reporting classification branch is untested
P3 0 none

Findings

Correctness

  • [P1] examples/17-taskspawner-ci-remediation/taskspawner.yaml:48 — GitHub delivers check_run events for non-PR branch checks and documents that fork pushes can have an empty pull_requests array. In those valid cases ParseGitHubWebhook omits Branch and Number, but this manifest unconditionally renders both in branch, nameTemplate, and promptTemplate; missingkey=error makes Task construction fail and the handler return 500 for every matching delivery. Gate this workflow on PR association before rendering, or make the implementation and example safely handle check runs without linked PR metadata.
  • [P1] examples/17-taskspawner-ci-remediation/taskspawner.yaml:16 — The PR's own checkRunPayload fixture models completed GitHub Actions checks with sender.login equal to github-actions[bot], while this manifest excludes that sender. Because spawner-level excludeAuthors runs before the conclusion and check-name filters, the advertised lint/test/build failures are rejected before a Task can be created. Remove this exclusion and use the explicit checkName filters to scope the intended checks.
  • [P2] examples/17-taskspawner-ci-remediation/taskspawner.yaml:52ci-fix-{{.Number}}-{{.CheckName}} is identical for every run of the same check on a PR, so a later failure on another commit or rerun is silently deduplicated while the earlier Task exists, including its one-hour post-completion TTL. Include the check run's {{.ID}} in the name so redeliveries of one run collapse without suppressing distinct runs.

Tests

  • [P2] internal/webhook/handler.go:886 — The new check_run reporting classification is not added to the existing TestWebhookSourceKind table, so the claimed PR-reporting behavior is unpinned. Add cases for a linked check run (Number > 0pull-request) and an unlinked check run (Number == 0issue).

Key takeaways

  • Both independent review paths identified the example's sender exclusion as conflicting with its intended GitHub Actions workflow.

- Drop excludeAuthors from the example: a check_run sender is the CI app, and
  spawner-level excludeAuthors runs before filters, so it dropped every event
- Name the Task by check run ID so redeliveries collapse while reruns and new
  commits get their own Task, instead of naming by PR number
- Render PR-dependent template vars via index / if-index so check runs without a
  linked PR no longer fail Task construction under missingkey=error
- Add TestWebhookSourceKind cases for linked and unlinked check runs
- Add ExtractGitHubWorkItem test asserting Branch/Number are omitted when a
  check_run has no linked PR

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature Categorizes issue or PR as related to a new feature needs-actor needs-priority needs-triage release-note

Projects

None yet

2 participants