diff --git a/docs/index.md b/docs/index.md index c563b2f..34fc6e1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -37,6 +37,8 @@ processes for customization and maintenance. - {doc}`/reference/patch-acceptance-criteria` • {doc}`/reference/stable-patch-format` • {doc}`/how-to/source-code/send-patches` * - **Kernel development** - {doc}`/how-to/source-code/enable-source-repositories` • {doc}`/how-to/source-code/obtain-kernel-source-git` • {doc}`/how-to/develop-customise/build-kernel` • {doc}`/how-to/develop-customise/build-kernel-snap` • {doc}`/how-to/testing-verification/test-kernel-in-proposed` • {doc}`/explanation/ubuntu-linux-kernel-sources` +* - **Kernel release and maintenance** + - {doc}`/reference/kernel-workflow-playbook/kernel-release` • {doc}`/reference/kernel-workflow-playbook/kernel-rollback` * - **Kernel variants** - {doc}`/explanation/stable-release-updates` • {doc}`/explanation/post-release-updates` • {doc}`/reference/hwe-kernels` • {doc}`/reference/oem-kernels` • {doc}`/reference/ubuntu-kernels/` * - **Upload rights** diff --git a/docs/reference/index.md b/docs/reference/index.md index f3c66b2..06dff62 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -35,6 +35,15 @@ oem-kernels snap-lifecycle ``` +## Kernel release and maintenance + +```{toctree} +:maxdepth: 1 + +Release an SRU Kernel +Rollback a Kernel +``` + ## Privileges Understand the criteria and process to apply for Ubuntu kernel and DKMS package upload rights. diff --git a/docs/reference/kernel-workflow-playbook/kernel-release.md b/docs/reference/kernel-workflow-playbook/kernel-release.md new file mode 100644 index 0000000..baa9009 --- /dev/null +++ b/docs/reference/kernel-workflow-playbook/kernel-release.md @@ -0,0 +1,73 @@ +--- +myst: + html_meta: + description: "Expedite an SRU kernel release to updates using copy-package-kernel. Learn the workflow for kernel team preparation and Archive Admin execution steps." +--- + +# Releasing an SRU kernel + +If you need to expedite the release of a kernel build as part of the SRU cycle process but you are unable to get hold of a Kernel Archive Admin (AA), you can use the following recipe. + +## Prepare (Kernel) + +To release a kernel it must be in calling to be released via the `promote-to-updates` task. +Liaise with the Kernel Stable team to get the testing and {spellexception}`signoffs` into an appropriate state to cause the tracker, a Launchpad bug against kernel-sru-workflow project, to ask to release. + +We can then form a release command for the Archive Admins to execute. + +```{code-block} shell +./copy-package-kernel --from-route proposed --to-route updates --tracker +``` + +## Execute (Archive Admin) + +Kernels are promoted using the `copy-package-kernel` command from [ubuntu-archive-tools]. +This command makes use of the kernel-team databases to identify the source and destination for the copies. +It also has internal validation to confirm that the package collection in the destination pocket will be internally consistent by versions after the copies. + +The kernel team will bring the bones of a `copy-package-kernel` command for the required promotion for execution. + +First, check that the tracker provided with the command is requesting to be released. +There should be a task against `promote-to-updates` which should be in "Confirmed" state. + +- If this is _not_ the case, then this should be handed back to the kernel-team for resolution. +- If it is, assign that task to yourself, and move it to "In Progress". + +The supplied command can be safely run with the `-n` argument to see what it would do; you can also add the `–verbose` option to dump out the equivalent `copy-package` kernel commands for validation. + +```{terminal} +:user: user +:host: host + +/copy-package-kernel --from-route proposed --to-route updates --tracker 2127318 -n --verbose + +copy-tracker: 2127318 (focal:linux-iot) proposed updates + Versions: -final- -was- + main 5.4.0-1056.59 5.4.0-1055.58 + meta 5.4.0.1056.54 5.4.0.1055.53 + signed 5.4.0-1056.59 5.4.0-1055.58 + Copies: + linux-iot 5.4.0-1056.59 ppa:canonical-kernel-esm/ubuntu/proposed:Release -> ppa:ubuntu-esm/ubuntu/esm-infra-security:Release ... dry-run + copy-package -n --include-binaries --auto-approve \ + --from ppa:canonical-kernel-esm/ubuntu/proposed --from-suite focal \ + --to ppa:ubuntu-esm/ubuntu/esm-infra-security --to-suite focal \ + --version 5.4.0-1056.59 linux-iot + linux-meta-iot 5.4.0.1056.54 ppa:canonical-kernel-esm/ubuntu/proposed:Release -> ppa:ubuntu-esm/ubuntu/esm-infra-security:Release ... dry-run + copy-package -n --include-binaries --auto-approve \ + --from ppa:canonical-kernel-esm/ubuntu/proposed --from-suite focal \ + --to ppa:ubuntu-esm/ubuntu/esm-infra-security --to-suite focal \ + --version 5.4.0.1056.54 linux-meta-iot + linux-signed-iot 5.4.0-1056.59 ppa:canonical-kernel-esm/ubuntu/proposed:Release -> ppa:ubuntu-esm/ubuntu/esm-infra-security:Release ... dry-run + copy-package -n --include-binaries --auto-approve \ + --from ppa:canonical-kernel-esm/ubuntu/proposed --from-suite focal \ + --to ppa:ubuntu-esm/ubuntu/esm-infra-security --to-suite focal \ + --version 5.4.0-1056.59 linux-signed-iot +``` + +If you are happy with the output, rerun it with `-y` to execute it. +It is safe to run the command more than once as it is idempotent. +Running it a second time will confirm the copies have been accepted by Launchpad. + +When the copy completes external tooling should manage the state of `promote-to-updates` through to "Fix Released". + +[ubuntu-archive-tools]: https://code.launchpad.net/ubuntu-archive-tools diff --git a/docs/reference/kernel-workflow-playbook/kernel-rollback.md b/docs/reference/kernel-workflow-playbook/kernel-rollback.md new file mode 100644 index 0000000..f3467b2 --- /dev/null +++ b/docs/reference/kernel-workflow-playbook/kernel-rollback.md @@ -0,0 +1,90 @@ +--- +myst: + html_meta: + description: "Roll back a faulty kernel by reverting to an earlier version. Covers revert-kernels-to-spin and Archive Admin remove-package and copy-package commands." +--- + +# Kernel rollback + +When a kernel is found to be so bad that the only option is to withdraw it from the archive, the typical approach is to replace it with the previous kernel. +This will not fix anything for those who have already upgraded their kernel, but can further prevent other {spellexception}`upgraders` becoming affected. + +There have also been cases where upgrades are no longer possible with an update and reverting the update can restore the ability to upgrade. + +This recipe will guide you through identifying the kernel versions to rollback to, and how to produce a recipe for a member of the Archive Admins (AA) to follow to perform the required rollback. + +(ref-kernel-workflow-playbook-rollback-tooling)= + +## Prerequisites + +```{include} /reuse/kernel-workflow-playbook-tooling.txt +``` + +## Prepare (Kernel) + +In order to revert the kernel in a pocket we need to identify an earlier version of a good kernel. +We typically identify this via a previous cycle or spin number, and handle. + +We want to remove any existing kernel package publications for this handle, and then copy back earlier publications. +Use the `revert-kernels-to-spin` command to generate AA commands to effectuate these: + +```{code-block} shell +/revert-kernels-to-spin --spin s2025.09.15 --handle noble:linux \ + --pocket updates --reason "Causing upgrade issues" --yes +``` + +## Validation (Kernel) + +The kernel team should review the versions that the revert is settled on, as shown in the revert output. +It is vital to confirm that any LRM or signed respins have been included. +Where there is a later version the tooling will emit a warning as below. +The versions should be updated manually in this case. + +```{code-block} shell +# jammy:linux-azure: spin=s2025.10.13-2 full_versions={'lrm': '5.15.0-1102.111+1', 'main': '5.15.0-1102.111', 'meta': '5.15.0.1102.100', 'signed': '5.15.0-1102.111'} +[...] +# WARNING: linux-restricted-modules-azure looks to have a repin not in the spin (5.15.0-1102.111+1) +``` + +## Execute (Archive Admins) + +In order to revert a kernel, all of the packages which make up a kernel (e.g. `linux`, `linux-signed`, `linux-meta`, `linux-restricted-modules` etc) must be reverted together. +The kernel team will identify these packages and the versions of which are faulty, and the older package versions which should be reinstated. +They will use {ref}`kernel tooling ` to generate `remove-package` and `copy-package` commands to roll-back the published versions of these packages. + +These will consist of two groups of commands: an initial set of removals, one per package, plus a second set of copies for the same packages. +While it is possible for the two sets to differ, additional consideration is necessary in this case. +For example: + +```{code-block} shell +remove-package linux --version 6.8.0-88.89 --archive ubuntu --suite noble-updates --removal-comment='Causing upgrade issues' -y +remove-package linux-meta --version 6.8.0-88.89 --archive ubuntu --suite noble-updates --removal-comment='Causing upgrade issues' -y +remove-package linux-restricted-modules --version 6.8.0-88.89+1 --archive ubuntu --suite noble-updates --removal-comment='Causing upgrade issues' -y +remove-package linux-restricted-signatures --version 6.8.0-88.89+1 --archive ubuntu --suite noble-updates --removal-comment='Causing upgrade issues' -y +remove-package linux-signed --version 6.8.0-88.89 --archive ubuntu --suite noble-updates --removal-comment='Causing upgrade issues' -y +copy-package linux --version 6.8.0-87.88 --from ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y +copy-package linux-meta --version 6.8.0-87.88 --from ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y +copy-package linux-restricted-modules --version 6.8.0-87.88+1 --from ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y +copy-package linux-restricted-signatures --version 6.8.0-87.88+1 --from ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y +copy-package linux-signed --version 6.8.0-87.88 --from ubuntu --from-suite noble-updates --include-binaries --force-same-destination --auto-accept -y +``` + +```{note} +If the `security` pocket is later than the newly rolled-back kernel version in `updates`, the same procedure should be applied to the `security` pocket. +``` + +## Execute (IS) + +Where the affected series include those in ESM, removals from the primary PPAs the packages will also need removing from the {spellexception}`repropro` repository. +Take a list of the removed packages in the ESM series to mattermost \~IS channel, and request for them to be removed. + +A sample removal command is shown below: + +```shell +reprepro --basedir /srv/esm-archive/fips-updates/reprepro/ removesrc \ + focal-infra-security openssh '1:9.6p1-3ubuntu13.7+Fips1' +``` + +```{tip} +If you are unable to contact the Canonical IS team directly, liaise with a member of the Canonical Kernel team to request the removal of said packages. +``` diff --git a/docs/reuse/kernel-workflow-playbook-tooling.txt b/docs/reuse/kernel-workflow-playbook-tooling.txt new file mode 100644 index 0000000..af47fa0 --- /dev/null +++ b/docs/reuse/kernel-workflow-playbook-tooling.txt @@ -0,0 +1,9 @@ +Install the tooling needed for the various kernel workflow playbook items: + +```{code-block} shell +pipx install ckt_workflow@git+https://git.launchpad.net/~apw/+git/ckt_workflow@latest +``` + +```{note} +The repository location for this project is subject to change. +``` \ No newline at end of file