doc: add decommissioning guide

[elijahgreenstein]

This PR adds a decommissioning guide to comply with SSDLC requirements.

This guide is based off of the guide to LXD introduced in canonical/lxd#18507, but with several changes:

1. The structure is reorganized: this guide is largely organized around decommissioning an entire cluster. A paragraph at the beginning of the guide instructs user wishing to decommission a single cluster member to first remove the cluster, and provides cross references to relevant pages in the LXD documentation. This paragraph then directs users to proceed to destroy the data on the removed member after removing the cluster.
2. Deleting replicators and cluster links is moved into the "Delete data" section, in order to follow the note about projects and the need to use the --project flag in order to delete project-scoped entities.
3. microcloud commands are used wherever possible, in place of lxc commands.
4. References to MicroCeph logging is included.
5. Details about removing MicroCeph OSDs is included.
6. Instructions about removing storage buckets is removed from this guide, on the assumption that MicroCloud users would not use Ceph Object buckets.
7. Cluster removal includes details about removing the Ceph monmap when reducing the cluster down to one member.
8. Snap removal covers MicroCloud, LXD, MicroCeph, and MicroOVN.
9. A note about ACME services is included, alongside the note about OIDC providers, in the final section.

Checklist

• I have read the contributing guidelines and attest that all commits in this PR are signed off, cryptographically signed, and follow this project's commit structure.
• I have checked and added or updated relevant documentation.

[elijahgreenstein]

@mseralessandri , @roosterfish -- I added the section on MicroCeph OSDs based on my reading of the MicroCloud guide to removing cluster members, which included a reference to the MicroCeph guide to removing disks. However, would these steps be necessary when decommissioning an entire cluster?

[roosterfish]

Thanks, left a few comments.

[roosterfish]

If we recommend updating the cluster certificate in the reamining cluster for LXD, likely we should also propose doing this for MicroCloud, MicroCeph and MicroOVN (Microclusters).

@mseralessandri I wonder if we should grow support in MicroCloud to generally trigger this across LXD and Micro* from a single MicroCloud command like microcloud cluster update-certificate

[roosterfish]

It seems MicroCeph has a microceph certificates command used to set RGW cert. Also MicroOVN comes with a microovn certificates command but it also seems to be used solely for OVN services, not Microcluster.

So it seems this is a general gap currently in all Micro* products to be able to rotate the underlying Microcluster certificate with straightforward commands. This is supported by Microcluster but not directly exposed.
In case a user would hit this today, we can instruct using curl against the Microcluster API though.

[roosterfish]

I have added #1453 with a note to update this guide.

[roosterfish]

This is not true. LXD blocks the pool deletion because the default profile has a disk device which references the pool. You can remove this device which will then allow you to remove the pool:

root@micro01:~# lxc storage rm remote
Error: The storage pool is currently in use
root@micro01:~# lxc profile show default
name: default
description: Default LXD profile
config: {}
devices:
  eth0:
    name: eth0
    network: default
    type: nic
  root:
    path: /
    pool: remote
    type: disk
used_by: []
project: default
root@micro01:~# lxc profile device rm default root
root@micro01:~# lxc storage rm remote

[roosterfish]

Any reason for not using a MicroCeph command directly?

root@micro01:~# microceph disk list
Disks configured in MicroCeph:
+-----+----------+----------------------------------------------------+
| OSD | LOCATION |                        PATH                        |
+-----+----------+----------------------------------------------------+
| 1   | micro01  | /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk2 |
+-----+----------+----------------------------------------------------+

There is also a --host-only flag to return only OSDs configured on this member (requires running the command on the to be decommissioned member).

[roosterfish]

Should we instead link to the doc page where those commands are already listed? Just to prevent duplicating this.

[roosterfish]

Worth adding a command that MicroCeph and MicroOVN might not be installed if the user chose to deploy a MicroCloud without those components?

[roosterfish]

Worth running microcloud status on any of the remaining members to verify everything got cleaned up and the member is gone?