kernel-workflow-playbook: kernel SRU release and kernel rollback guides

docs/index.md

@@ -37,6 +37,8 @@ processes for customization and maintenance.
   - {doc}`/reference/patch-acceptance-criteria` • {doc}`/reference/stable-patch-format` • {doc}`/how-to/source-code/send-patches`
 * - **Kernel development**
   - {doc}`/how-to/source-code/enable-source-repositories` • {doc}`/how-to/source-code/obtain-kernel-source-git` • {doc}`/how-to/develop-customise/build-kernel` • {doc}`/how-to/develop-customise/build-kernel-snap` • {doc}`/how-to/testing-verification/test-kernel-in-proposed` • {doc}`/explanation/ubuntu-linux-kernel-sources`
+* - **Kernel release and maintenance**
+  - {doc}`/reference/kernel-workflow-playbook/kernel-release` • {doc}`/reference/kernel-workflow-playbook/kernel-rollback`
 * - **Kernel variants**
   - {doc}`/explanation/stable-release-updates` • {doc}`/explanation/post-release-updates` • {doc}`/reference/hwe-kernels` • {doc}`/reference/oem-kernels` • {doc}`/reference/ubuntu-kernels/`
 * - **Upload rights**

docs/reference/index.md

@@ -35,6 +35,15 @@ oem-kernels
 snap-lifecycle
 ```
 
+## Kernel release and maintenance
+
+```{toctree}
+:maxdepth: 1
+
+Release an SRU Kernel </reference/kernel-workflow-playbook/kernel-release>
+Rollback a Kernel </reference/kernel-workflow-playbook/kernel-rollback>
+```
+
 ## Privileges
 
 Understand the criteria and process to apply for Ubuntu kernel and DKMS package upload rights.

docs/reference/kernel-workflow-playbook/kernel-release.md

@@ -0,0 +1,73 @@
+---
+myst:
+  html_meta:
+    description: "Expedite an SRU kernel release to updates using copy-package-kernel. Learn the workflow for kernel team preparation and Archive Admin execution steps."
+---
+
+# Releasing an SRU kernel
+
+If you need to expedite the release of a kernel build as part of the SRU cycle process but you are unable to get hold of a Kernel Archive Admin (AA), you can use the following recipe.
+
+## Prepare (Kernel)
+
+To release a kernel it must be in calling to be released via the `promote-to-updates` task. 
+Liaise with the Kernel Stable team to get the testing and {spellexception}`signoffs` into an appropriate state to cause the tracker, a Launchpad bug against kernel-sru-workflow project, to ask to release.
+
+We can then form a release command for the Archive Admins to execute.
+
+```{code-block} shell
+./copy-package-kernel --from-route proposed --to-route updates --tracker <tracker>
+```
+
+## Execute (Archive Admin)
+
+Kernels are promoted using the `copy-package-kernel` command from [ubuntu-archive-tools].
+This command makes use of the kernel-team databases to identify the source and destination for the copies.
+It also has internal validation to confirm that the package collection in the destination pocket will be internally consistent by versions after the copies.
+
+The kernel team will bring the bones of a `copy-package-kernel` command for the required promotion for execution. 
+
+First, check that the tracker provided with the command is requesting to be released.
+There should be a task against `promote-to-updates` which should be in "Confirmed" state.
+
+- If this is _not_ the case, then this should be handed back to the kernel-team for resolution.
+- If it is, assign that task to yourself, and move it to "In Progress".
+
+The supplied command can be safely run with the `-n` argument to see what it would do; you can also add the `–verbose` option to dump out the equivalent `copy-package` kernel commands for validation.
+
+```{terminal}
+:user: user
+:host: host
+
+/copy-package-kernel --from-route proposed --to-route updates --tracker 2127318 -n --verbose
+
+copy-tracker: 2127318 (focal:linux-iot) proposed updates
+  Versions:   -final-        -was-
+    main      5.4.0-1056.59  5.4.0-1055.58
+    meta      5.4.0.1056.54  5.4.0.1055.53
+    signed    5.4.0-1056.59  5.4.0-1055.58
+  Copies:
+    linux-iot                                     5.4.0-1056.59             ppa:canonical-kernel-esm/ubuntu/proposed:Release -> ppa:ubuntu-esm/ubuntu/esm-infra-security:Release ... dry-run
+      copy-package -n --include-binaries --auto-approve \
+        --from ppa:canonical-kernel-esm/ubuntu/proposed	 --from-suite focal \
+        --to ppa:ubuntu-esm/ubuntu/esm-infra-security --to-suite focal \
+        --version 5.4.0-1056.59 linux-iot
+    linux-meta-iot                                5.4.0.1056.54             ppa:canonical-kernel-esm/ubuntu/proposed:Release -> ppa:ubuntu-esm/ubuntu/esm-infra-security:Release ... dry-run
+      copy-package -n --include-binaries --auto-approve \
+        --from ppa:canonical-kernel-esm/ubuntu/proposed --from-suite focal \
+        --to ppa:ubuntu-esm/ubuntu/esm-infra-security --to-suite focal \
+        --version 5.4.0.1056.54 linux-meta-iot
+    linux-signed-iot                              5.4.0-1056.59             ppa:canonical-kernel-esm/ubuntu/proposed:Release -> ppa:ubuntu-esm/ubuntu/esm-infra-security:Release ... dry-run
+      copy-package -n --include-binaries --auto-approve \
+        --from ppa:canonical-kernel-esm/ubuntu/proposed --from-suite focal \
+        --to ppa:ubuntu-esm/ubuntu/esm-infra-security --to-suite focal \
+        --version 5.4.0-1056.59 linux-signed-iot
+```
+
+If you are happy with the output, rerun it with `-y` to execute it.
+It is safe to run the command more than once as it is idempotent.
+Running it a second time will confirm the copies have been accepted by Launchpad. 
+
+When the copy completes external tooling should manage the state of `promote-to-updates` through to "Fix Released".
+
+[ubuntu-archive-tools]: https://code.launchpad.net/ubuntu-archive-tools

docs/reference/kernel-workflow-playbook/kernel-rollback.md

@@ -0,0 +1,90 @@
+---
+myst:
+  html_meta:
+    description: "Roll back a faulty kernel by reverting to an earlier version. Covers revert-kernels-to-spin and Archive Admin remove-package and copy-package commands."
+---
+
+# Kernel rollback
+
+When a kernel is found to be so bad that the only option is to withdraw it from the archive, the typical approach is to replace it with the previous kernel.
+This will not fix anything for those who have already upgraded their kernel, but can further prevent other {spellexception}`upgraders` becoming affected.
+
+There have also been cases where upgrades are no longer possible with an update and reverting the update can restore the ability to upgrade.
+
+This recipe will guide you through identifying the kernel versions to rollback to, and how to produce a recipe for a member of the Archive Admins (AA) to follow to perform the required rollback.
+
+(ref-kernel-workflow-playbook-rollback-tooling)=
+
+## Prerequisites
+
+```{include} /reuse/kernel-workflow-playbook-tooling.txt
+```
+
+## Prepare (Kernel)
+
+In order to revert the kernel in a pocket we need to identify an earlier version of a good kernel.
+We typically identify this via a previous cycle or spin number, and handle.
+
+We want to remove any existing kernel package publications for this handle, and then copy back earlier publications. 
+Use the `revert-kernels-to-spin` command to generate AA commands to effectuate these:
+
+```{code-block} shell
+/revert-kernels-to-spin --spin s2025.09.15 --handle noble:linux \
+        --pocket updates --reason "Causing upgrade issues" --yes
+```
+
+## Validation (Kernel)
+
+The kernel team should review the versions that the revert is settled on, as shown in the revert output.
+It is vital to confirm that any LRM or signed respins have been included.
+Where there is a later version the tooling will emit a warning as below.
+The versions should be updated manually in this case.
+
+```{code-block} shell
+# jammy:linux-azure: spin=s2025.10.13-2 full_versions={'lrm': '5.15.0-1102.111+1', 'main': '5.15.0-1102.111', 'meta': '5.15.0.1102.100', 'signed': '5.15.0-1102.111'}
+[...]
+# WARNING: linux-restricted-modules-azure looks to have a repin not in the spin (5.15.0-1102.111+1)
+```
+
+## Execute (Archive Admins)
+
+In order to revert a kernel, all of the packages which make up a kernel (e.g. `linux`, `linux-signed`, `linux-meta`, `linux-restricted-modules` etc) must be reverted together.
+The kernel team will identify these packages and the versions of which are faulty, and the older package versions which should be reinstated.
+They will use {ref}`kernel tooling <ref-kernel-workflow-playbook-rollback-tooling>` to generate `remove-package` and `copy-package` commands to roll-back the published versions of these packages.
+
+These will consist of two groups of commands: an initial set of removals, one per package, plus a second set of copies for the same packages.
+While it is possible for the two sets to differ, additional consideration is necessary in this case.
+For example:
+
+```{code-block} shell
+remove-package linux                       --version 6.8.0-88.89   --archive ubuntu --suite      noble-updates --removal-comment='Causing upgrade issues' -y
+remove-package linux-meta                  --version 6.8.0-88.89   --archive ubuntu --suite      noble-updates --removal-comment='Causing upgrade issues' -y
+remove-package linux-restricted-modules    --version 6.8.0-88.89+1 --archive ubuntu --suite      noble-updates --removal-comment='Causing upgrade issues' -y
+remove-package linux-restricted-signatures --version 6.8.0-88.89+1 --archive ubuntu --suite      noble-updates --removal-comment='Causing upgrade issues' -y
+remove-package linux-signed                --version 6.8.0-88.89   --archive ubuntu --suite      noble-updates --removal-comment='Causing upgrade issues' -y
+copy-package   linux                       --version 6.8.0-87.88   --from    ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y
+copy-package   linux-meta                  --version 6.8.0-87.88   --from    ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y
+copy-package   linux-restricted-modules    --version 6.8.0-87.88+1 --from    ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y
+copy-package   linux-restricted-signatures --version 6.8.0-87.88+1 --from    ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y
+copy-package   linux-signed                --version 6.8.0-87.88   --from    ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y
+```
+
+```{note}
+If the `security` pocket is later than the newly rolled-back kernel version in `updates`, the same procedure should be applied to the `security` pocket.
+```
+
+## Execute (IS)
+
+Where the affected series include those in ESM, removals from the primary PPAs the packages will also need removing from the {spellexception}`repropro` repository.
+Take a list of the removed packages in the ESM series to mattermost \~IS channel, and request for them to be removed.
+
+A sample removal command is shown below:
+
+```shell
+reprepro --basedir /srv/esm-archive/fips-updates/reprepro/ removesrc \
+      focal-infra-security openssh '1:9.6p1-3ubuntu13.7+Fips1'
+```
+
+```{tip}
+If you are unable to contact the Canonical IS team directly, liaise with a member of the Canonical Kernel team to request the removal of said packages.
+```

docs/reuse/kernel-workflow-playbook-tooling.txt

@@ -0,0 +1,9 @@
+Install the tooling needed for the various kernel workflow playbook items:
+
+```{code-block} shell
+pipx install ckt_workflow@git+https://git.launchpad.net/~apw/+git/ckt_workflow@latest
+```
+
+```{note}
+The repository location for this project is subject to change.
+```