feat: add Prow CI health alerts - #6474
Conversation
|
Skipping CI for Draft Pull Request. |
There was a problem hiding this comment.
Pull request overview
This PR adds Azure Monitor Prometheus alert rules to proactively detect degraded ARO-HCP Prow CI health (persistent job failures, degraded regional provision healthchecks, slow e2e-parallel successful runs, stale/malformed collection, and blocked Tide/batch activity). It also extends the existing Prow metrics collector to export a “consecutive failures” signal needed for a batch-streak alert.
Changes:
- Export a new
prow_ci_job_consecutive_failuresmetric computed from completed run history. - Add a dedicated
prow-ci-alertsPrometheus rule group with multiple CI-health alert rules and thresholds. - Add unit test coverage validating consecutive-failure ordering behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tooling/tenant-quota/pkg/prow/collector.go | Adds per-(job_name, job_type) consecutive failure streak metric emitted by the Prow collector. |
| tooling/tenant-quota/pkg/prow/collector_test.go | Adds unit test validating failure streak calculation uses completion time ordering. |
| tooling/tenant-quota/alerting.bicep | Introduces prow-ci-alerts Prometheus rule group with CI health alerts (permafailing, healthcheck rate, duration P95, stale collection, invalid jobs, batch streak). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Derive per-job failure streaks from completion-ordered cached runs so Tide batch queue blockage can be alerted without per-build timestamp metrics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Alert on permafailing jobs, regional healthcheck success, e2e duration, stale or malformed collection data, empty caches, and blocked Tide merge queues. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
9ee8fba to
9d0d644
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
tooling/tenant-quota/pkg/prow/collector.go:256
- The tie-breaker in the consecutive-failures sort compares
buildIDlexicographically. If two runs have the samecompletionAt(possible at second-level timestamp resolution), lexicographic ordering can misorder numeric build IDs (e.g., "9" sorts after "10"), which can produce an incorrect failure streak.
Consider using a numeric-string ordering for buildID (length then lexicographic) to keep the ordering consistent with numeric build IDs without needing per-compare parsing.
slices.SortFunc(jobRuns, func(a, b runMetrics) int {
if order := b.completionAt.Compare(a.completionAt); order != 0 {
return order
}
return strings.Compare(b.buildID, a.buildID)
})
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
tooling/tenant-quota/pkg/prow/collector.go:256
- The tie-breaker in the sort for consecutive failure streaks uses
strings.CompareonbuildID. Prow build IDs are typically numeric strings, and lexicographic ordering can be wrong when lengths differ (e.g., "9" sorts after "10"), which can misorder runs that share the samecompletionAttimestamp and produce an incorrect consecutive-failure count.
slices.SortFunc(jobRuns, func(a, b runMetrics) int {
if order := b.completionAt.Compare(a.completionAt); order != 0 {
return order
}
return strings.Compare(b.buildID, a.buildID)
|
/retest |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hbhushan3, roivaz The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Why
ARO-HCP Prow health currently requires manual inspection, which delays detection of persistently failing jobs, degraded regional provision healthchecks, slow e2e runs, stale telemetry, and a blocked Tide merge queue.
This is a follow-up to #6374. There is no separate tracking ticket because these alert thresholds and failure modes were defined during review of that telemetry work.
What
e2e-parallelsuccessful-run P95 above 2h30mprow_ci_job_consecutive_failuresbecause an ordered failure streak cannot be derived from the existing per-run metric series.Live baseline
Evaluated against the opstool Azure Monitor workspace on 2026-08-10. Only
ProwCIScheduledJobPermafailingwould fire, for these four periodic cleanup jobs:periodic-ci-azure-aro-hcp-main-periodic-cleanup-delete-expired-integration-resource-groupsperiodic-ci-azure-aro-hcp-main-periodic-cleanup-delete-expired-prod-00-resource-groupsperiodic-ci-azure-aro-hcp-main-periodic-cleanup-delete-expired-prod-resource-groupsperiodic-ci-azure-aro-hcp-main-periodic-cleanup-delete-expired-stage-resource-groupsThe regional provision healthcheck is at 91.7% success and
e2e-parallelsuccessful-run P95 is 2h26m30s, so neither crosses its threshold. Collection is healthy with 839 cached runs and no malformed jobs in the last hour. The undeployed Tide batch-streak rule is excluded from this live evaluation.Validation
go test -race ./pkg/prow/...golangci-lintaz bicep build --file tooling/tenant-quota/alerting.bicep