Skip to content

docs: add DEV CI regional failover runbook - #6510

Draft
roi (roivaz) wants to merge 1 commit into
Azure:mainfrom
roivaz:aro-27246-27248-dev-region-failover
Draft

docs: add DEV CI regional failover runbook#6510
roi (roivaz) wants to merge 1 commit into
Azure:mainfrom
roivaz:aro-27246-27248-dev-region-failover

Conversation

@roivaz

Copy link
Copy Markdown
Collaborator

ARO-27246
ARO-27248

What

Add an actionable DEV CI regional failover and failback runbook covering:

  • provision-health drain, fallback eligibility, and re-enable criteria
  • the e2e-parallel, e2e-parallel-inplace-upgrade, and /test capz-e2e-dev region switches
  • prerequisites, decision points, release-config procedure, validation, rollback, ownership, and escalation
  • the boundary between automated health signals and required manual operator actions

Link the runbook from the CI overview, monitoring, and operations documentation.

Why

DEV CI already probes westus3, centralus, and canadacentral and alerts on sustained provision failure, but it does not automatically drain or re-enable a region. Operators need one technically validated procedure that turns those signals into a consistent manual decision and updates every affected presubmit together.

Testing

Documentation-only change; no product tests were added.

  • Rendered the dev/ci01 partial configuration successfully for westus3, centralus, and canadacentral.
  • Validated relative Markdown links in all changed files.
  • Ran git diff --check.
  • Confirmed the repository has no dedicated documentation check target.
  • Cross-checked alert thresholds and retention against ARO-HCP config and the live job names, variables, schedules, workflows, and generated Prow config against current openshift/release origin/main.

Special notes for your reviewer

This PR is intentionally opened as a draft. Reviewers have not been requested yet.

PR Checklist

  • PR is scoped to a single task (no mixed concerns)
  • Title follows Conventional Commits format
  • Summary explains the "Why" behind the change
  • Linked to relevant ticket/issue
  • Screenshots included (if graph/UI/metrics changes) — not applicable; documentation only
  • Self-reviewed the diff
  • CI/CD checks are passing (ignore Tide)
  • Draft PR used for WIP (if applicable)
  • Commit history is clean (rebased/squashed)
  • Tricky code blocks are commented — not applicable; documentation only
  • Specific reviewers tagged — intentionally deferred while draft
  • All comment threads resolved before merge

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: roivaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant