From a41c89b30b4bb95602f9317e3d27a0ccaf96be3f Mon Sep 17 00:00:00 2001 From: Elijah Greenstein Date: Tue, 7 Jul 2026 15:06:27 -0700 Subject: [PATCH] doc: add decommissioning guide Signed-off-by: Elijah Greenstein --- doc/how-to/decommission.md | 266 ++++++++++++++++++++++++++++++++++++ doc/how-to/index.md | 1 + doc/how-to/member_remove.md | 1 + 3 files changed, 268 insertions(+) create mode 100644 doc/how-to/decommission.md diff --git a/doc/how-to/decommission.md b/doc/how-to/decommission.md new file mode 100644 index 000000000..31c29de6c --- /dev/null +++ b/doc/how-to/decommission.md @@ -0,0 +1,266 @@ +--- +myst: + html_meta: + description: Follow these steps to securely decommission a MicroCloud cluster member or cluster. +--- + +(howto-decommission)= +# How to securely decommission a MicroCloud deployment + +```{important} +This process will erase all data associated with your MicroCloud deployment. +Make copies of any data that you need to preserve before proceeding. +Refer to {ref}`lxd:instances-backup` and {ref}`lxd:howto-storage-backup-volume` for relevant details. +``` + +This guide walks you through the steps to decommission an entire MicroCloud cluster. + +If you only need to decommission a single cluster member, first {ref}`remove the member from the cluster `. +After removing the member, {ref}`update the certificate ` on the cluster remaining in production. +Then, return to this guide and skip ahead to {ref}`howto-decommission-destroy-data` for additional details. + +Some commands used to decommission MicroCloud are LXD or MicroCeph commands. +Refer to the {ref}`LXD decommissioning guide ` and the {ref}`MicroCeph guide to removing disks ` for related information. + +(howto-decommission-remove-offline-member)= +## Remove offline cluster members + +Use the `--force` flag to {ref}`remove any offline cluster members ` (you will remove online cluster members later in the process): + +```bash +sudo microcloud remove --force +``` + +(howto-decommission-revoke-remote)= +## Revoke remote access + +List all identities that have access to LXD, then delete each identity: + +```bash +lxc auth identity list +lxc auth identity delete / +``` + +(howto-decommission-delete-data)= +## Delete data + +You can run the commands in this section on any cluster member to delete data. + +```{important} +Data deleted by LXD physically remains on disks and can be recovered by users with access to the disks. +To prevent unauthorized data recovery, you must {ref}`destroy and sanitize your data `. +``` + +### List projects + +Replicators, instances, profiles, and custom volumes are scoped by {ref}`project `. +For deployments with more than one project, you must repeat some steps for **each** project, each time using the `--project` flag. +You do not need to use the `--project` flag to decommission deployments with only one project. + +Run this command to get a list of all projects: + +```bash +lxc project list +``` + +````{note} +You can also delete a project (except the `default` project) and all of its project-level entities with: + +```bash +lxc project delete --force +``` +```` + +### Delete replicators and cluster links + +For each project, list all replicators, then delete each replicator: + +```bash +lxc replicator list --project +lxc replicator delete --project +``` + +Likewise, list all cluster links, then delete each cluster link (cluster links are not scoped by project so you do not need to use the `--project` flag): + +```bash +lxc cluster link list +lxc cluster link delete +``` + + +### Stop and delete instances + +For each project, stop all instances: + +```bash +lxc stop --all --project +``` + +Next, for each project, list all instances, then delete each instance: + +```bash +lxc list --project +lxc delete --project +``` + +````{note} +If you are unable to stop or delete an instance, use the `--force` flag: + +```bash +lxc stop --force --project +lxc delete --force --project +``` +```` + +### Delete profiles + +List all profiles per project, then delete every profile but the `default` profile (the `default` profile cannot be deleted): + +```bash +lxc profile list --project +lxc profile delete --project +``` + +### Delete custom volumes + +To delete {ref}`custom volumes `, you must specify the storage pools used by the volumes. +First, list all storage pools across projects: + +```bash +lxc storage list +``` + +Next, for each storage pool, list the custom volumes. +Use the `--all-projects` flag to view all custom volumes across projects: + +```bash +lxc storage volume list type=custom --all-projects +``` + +Use the `PROJECT` column in the output to identify the project associated with each custom volume. +Then delete each custom volume, specifying both the storage pool and the project: + +```bash +lxc storage volume delete --project +``` + +### Delete storage pools + +Storage pools cannot be deleted if they are used by an instance, profile, or custom volume. +The `default` profile cannot be deleted; therefore, the storage pool used by the `default` profile cannot be deleted. +To identify this storage pool, view information about the `default` profile and find the pool listed under `devices` > `root` > `pool`: + +```bash +lxc profile show default +``` + +Next, list all storage pools, then delete every pool but the one used by the `default` profile. + +```bash +lxc storage list +lxc storage delete +``` + +```{note} +You do not need to specify a project when running these commands. +``` + +### Delete monitoring data + +Delete data from any external systems that you used to monitor {ref}`LXD events `, {ref}`LXD metrics `, or {ref}`Ceph logging `, such as [Loki](https://grafana.com/oss/loki/), [Prometheus](https://prometheus.io/), or [Grafana](https://grafana.com/). +Refer to the documentation for those systems for details. + + +(howto-decommission-remove-microceph-osds)= +## Remove MicroCeph OSDs + +To {ref}`remove MicroCeph OSDs `, you need to determine the OSD IDs. +Run this command to view the Ceph OSD tree: + +``` +ceph osd tree +``` + +Identify the OSD IDs under each host, then remove each OSD: + +``` +sudo microceph disk remove +``` + +````{note} +If you are unable to remove an OSD, use the `--bypass-safety-checks` flag: + +```bash +sudo microceph disk remove --bypass-safety-checks +``` +```` + +Finally, verify that the OSDs have been removed: + +``` +ceph osd tree +``` + +(howto-decommission-remove-remaining-members)= +## Remove remaining cluster members + +After deleting data, you can remove the online cluster members from the cluster. + +Then list all cluster members and remove every member except the member on which you are running these commands: + +```bash +microcloud cluster list +sudo microcloud remove +``` + +````{note} +To {ref}`reduce the cluster down to one member `, you must first clean up the Ceph monitor map (`monmap`) while there are still two cluster members: + +```bash +sudo microceph.ceph mon remove +sudo microceph cluster sql "delete from services where member_id = (select id from core_cluster_members where name='') and service='mon'" +``` +```` + +(howto-decommission-remove-microcloud)= +## Remove snaps + +```{important} +Run these commands on **every** machine that you decommission. + +Removing MicroCloud **does not** erase {ref}`ZFS pools (zpools) ` or dedicated disks used by MicroCeph as Ceph object storage daemons (OSDs). +To securely decommission MicroCloud, you must {ref}`destroy and sanitize your data `. +``` + +Remove the MicroCloud, LXD, MicroCeph, and MicroOVN snaps. +Use the `--purge` flag, or a snapshot of your data will be preserved: + +```bash +sudo snap remove microcloud --purge +sudo snap remove lxd --purge +sudo snap remove microceph --purge +sudo snap remove microovn --purge +``` + +Verify that the snaps and associated data were removed. +The following commands should report that none of these snaps are installed and that the `/var/snap/microcloud/`, `/var/snap/lxd/`, `/var/snap/microceph/`, and `/var/snap/microovn` directories do not exist: + +```bash +snap list microcloud lxd microceph microovn +ls /var/snap/microcloud/ /var/snap/lxd/ /var/snap/microceph/ /var/snap/microovn/ +``` + +(howto-decommission-destroy-data)= +## Destroy and sanitize data + +Data deleted with MicroCloud, LXD, or Ceph commands remains readable and can be recovered by users with access to disks used in your deployment. +To prevent unauthorized recovery, you must physically overwrite the data. +Follow your data destruction policy to securely erase or destroy the disks that you are decommissioning. + +If you are decommissioning an entire MicroCloud, apply your data destruction policy to any machines used to monitor events, logs, or metrics. +For clusters {ref}`configured with OIDC `, consult your OIDC identity provider for the steps to remove any data associated with your profile. +Likewise, if you used {ref}`ACME services to issue server certificates `, refer to the service provider for the steps to remove any associated data. + +```{important} +Sanitized data is irreversibly destroyed and cannot be recovered. +``` diff --git a/doc/how-to/index.md b/doc/how-to/index.md index 83f476a28..d51c7907b 100644 --- a/doc/how-to/index.md +++ b/doc/how-to/index.md @@ -59,6 +59,7 @@ Manage multiple clusters Recover MicroCloud Update and upgrade Manage the snaps +Decommission MicroCloud ``` ## Engage with us diff --git a/doc/how-to/member_remove.md b/doc/how-to/member_remove.md index 499bc2833..0e5e1af99 100644 --- a/doc/how-to/member_remove.md +++ b/doc/how-to/member_remove.md @@ -31,6 +31,7 @@ If the machine is no longer reachable over the network, you can also add the `-- Removing a cluster member with `--force` will not attempt to perform any clean-up of the removed machine. All services will need to be fully re-installed before they can be re-initialized. Resources allocated to the MicroCloud like disks and network interfaces may need to be re-initialized as well. ``` +(howto-member-remove-reduce-cluster)= ## Reducing the cluster to one member When shrinking the cluster down to one member, you must also clean up the Ceph monitor map (`monmap`) before proceeding, even when using the `--force` flag.