ci: auto-create GitHub issues on nightly e2e failure

[R-Lawton]

Summary

• automatically create a GitHub issue when nightly e2e tests fail, with the list of failed tests and a link to the run
• classifies failures as infrastructure (setup/cluster) or test (playwright) and labels accordingly
• no duplicate issues — checks for existing open issues before creating
• issues left open for manual triage

Changes

• e2e-nightly.yaml — create-issue job files issues labelled nightly-failure/nightly-infrastructure + e2e
• e2e-common.yaml — added failure classification output (infrastructure vs test vs none) and JSON results artifact upload
• e2e/playwright.config.ts — added JSON reporter for machine-readable test results

Verification

Tested on fork with a deliberate failing test:

• Run 1 — issue created
• Run 2 — no duplicate issue created
• Auto-created issue

Test plan

• trigger via workflow_dispatch and verify the workflow runs successfully
• verify issue creation on test failure
• verify no duplicate issues are created

Closes #586

Summary by CodeRabbit

• New Features

  • Nightly end-to-end test failures now automatically create GitHub issues with relevant labels and failure details.
  • Test runs now produce JSON results alongside the existing HTML report.
  • Failure outcomes are classified as infrastructure failures, test failures, or successful runs.

• Bug Fixes

  • Improved failure reporting by including failed and flaky test names where available.
  • Duplicate nightly failure issues are avoided when an existing open issue is found.

[coderabbitai]

Warning

Review limit reached

@R-Lawton, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 52 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6f554cb2-e5c3-479f-ad98-63dff84efa3a

📥 Commits

Reviewing files that changed from the base of the PR and between 6e7afc8 and 3d24259.

📒 Files selected for processing (3)

• .github/workflows/e2e-common.yaml
• .github/workflows/e2e-nightly.yaml
• e2e/playwright.config.ts

📝 Walkthrough

Walkthrough

The reusable E2E workflow now classifies infrastructure and test failures, exposes the classification, and uploads JSON results. The nightly workflow consumes this output to extract failed tests and create deduplicated, labelled GitHub issues.

Changes

Nightly E2E reporting

Layer / File(s)	Summary
Failure classification and result publishing .github/workflows/e2e-common.yaml, e2e/playwright.config.ts	The reusable workflow identifies setup, server, and smoke/full test outcomes, exposes failure-type, uploads JSON results, and configures Playwright’s JSON reporter.
Nightly issue creation .github/workflows/e2e-nightly.yaml	The nightly workflow downloads available results, extracts failed or flaky test names, selects infrastructure or test labels, checks for existing open issues, and creates an issue when appropriate.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• Kuadrant/kuadrant-console-plugin#562: Introduced the reusable workflow and nightly workflow that these failure-reporting changes extend.
• Kuadrant/kuadrant-console-plugin#615: Overlaps with the smoke/full Playwright workflow changes in e2e-common.yaml.

Poem

I’m a rabbit with results in my paws,
Sorting test failures from cluster flaws.
JSON hops into the nightly stream,
While labelled issues guard the team’s dream.
No duplicate burrows shall appear—
E2E alerts now spring up clear!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	For [#586], the PR covers issue creation, deduping and labelling, but the summary does not show infra retries, a 4-night infrastructure issue, or auto-closing on recovery.	Add automatic retry handling for infra failures, create a separate nightly-infrastructure issue after four consecutive infra failures, and close the issue when nightly tests pass again.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately reflects the main change: automatic GitHub issue creation for nightly E2E failures.
Out of Scope Changes check	✅ Passed	The Playwright JSON reporter and result upload support the nightly issue-creation flow and do not appear to be out of scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[emmaaroche]

👀

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/e2e-common.yaml:
- Around line 94-105: Update the Classify failure step so its fallback branch
classifies any unhandled failure as infrastructure rather than none, while
preserving the existing tests/tests-full classification and successful-run
behavior. Ensure earlier install/startup step failures receive
type=infrastructure.

In @.github/workflows/e2e-nightly.yaml:
- Line 23: Update the checkout step using actions/checkout@v4 to disable
persisted credentials by setting persist-credentials to false; if the job has no
other need for repository files, remove the checkout step instead.
- Around line 8-10: Move the issues: write and actions: read permissions from
the workflow-level configuration into the create-issue job, leaving the e2e job
without those elevated permissions while preserving the create-issue job’s
access for gh issue, gh label, and artifact download operations.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0733fc1d-642b-4b75-be1c-a5de4f7b30da

📥 Commits

Reviewing files that changed from the base of the PR and between 09b1c97 and 6e7afc8.

📒 Files selected for processing (3)

• .github/workflows/e2e-common.yaml
• .github/workflows/e2e-nightly.yaml
• e2e/playwright.config.ts

[emmaaroche]

LGTM 👍