Add KB: Cluster Recovery After Full Power Outage (closes #441)#442
Open
CarlRodabaugh wants to merge 5 commits into
Open
Add KB: Cluster Recovery After Full Power Outage (closes #441)#442CarlRodabaugh wants to merge 5 commits into
CarlRodabaugh wants to merge 5 commits into
Conversation
Closes #441 Adds a consolidated how-to / troubleshooting guide for recovering a VergeOS cluster after an unplanned full power loss. Covers expected behavior, pre-power-on checks, the Node1 → Node2 → remaining-nodes sequence (including the "Waiting for the vSAN to mount" prompt as the operator's go-signal), post-recovery verification, troubleshooting (stuck repairs, split-brain, failed rejoin), prevention (UPS sizing, graceful shutdown automation via API/VRG, ioGuardian repair server, fencing handled internally), and when to engage support with sysdiag generation steps. Patterned after kb-template.md; cross-links to existing power-sequence, shutdown-procedure, journal-walks, repair-server, vSAN-diagnostics, sizing, and system-diagnostics docs.
- draft: false (frontmatter blocker) - Promote Prevention, When to Engage Support, Generating a System Diagnostic for Support from h3 to h2 - Drop unverified verbatim "Waiting for the vSAN to mount" console string; describe the wait-state behavior instead - Drop undefined "Kill Mode" term; describe IPMI hard power-off inline - Link Maintenance Mode to /product-guide/operations/maintenance-mode/ - Align slug to filename (cluster-recovery-after-power-outage) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…w fixes
- Replace 'at least 2 nodes' quorum framing with N-1 vSAN nodes (per Jason Yaeger).
- Simplify power-on sequence to 'Node1 first, then power on the rest paced ~1 min apart'; vSAN mounts on its own when N-1 is reached.
- Reframe Bad Drives: count of drives the cluster currently can't see; persistent non-zero is a real fault, not transient walk noise.
- Fix cluster shutdown API payload to include cluster id and params.
- Soften 'no built-in NUT/UPS' claim ('does not currently document').
- Drop unsupported 'often auto-created' claim about ioGuardian.
- Reframe split-brain as a recovery-time network-partition risk, not 'during the outage'.
- Trim Pro Tip; 'stoplights' -> 'status lights'; bump last-updated.
bcampbellverg
left a comment
bcampbellverg
left a comment
Collaborator
There was a problem hiding this comment.
multiple changes to the initial new KB article:
- stronger encouragement to prevent abrupt power loss to a cluster when possible,
- change power on sequence instructions-- no need to wait 1 minute between each node,
- added additional information about post-power-on verifications,
- modified vague language "before any destructive action" to more specifically say "before rebooting nodes or making significant changes"
- Changed "Prevention" section to "Prevention/Mitigation" because many of the items listed are intended to mitigate issues that can be caused by ungraceful shutdown, rather than prevent the situation.
Modified existing kb article for proper power on procedures:
- change power on sequence instructions-- no need to wait 1 minute between each node
- claude grammar changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new KB article addressing #441: a consolidated how-to / troubleshooting guide for recovering a VergeOS cluster after an unplanned full power loss.
Changes
docs/knowledge-base/posts/cluster-recovery-after-power-outage.mdkb-template.md(frontmatter, Key Points, Prerequisites, Steps, Troubleshooting, Prevention, Additional Resources, Feedback, Document Information)Issue coverage (per #441 "Suggested Content")
What to Expectsubsection)Waiting for the vSAN to mountconsole prompt as the operator's go-signal between Node1 and Node2support@verge.iofor the air-gapped/manual pathVerification
POST /v4/cluster_actions { action: shutdown }matches the existing Proper Shutdown Procedure doc and API Tables referenceTest plan
Waiting for the vSAN to mountconsole string matches what customers see on 26.x (verified against an in-the-wild console screenshot)