Skip to content

Commit 0212dd1

Browse files
JoeAldingerJoeAldinger
authored
Merge pull request #74713 from StephenJamesSmith/TELCODOCS-1820
TELCODOCS-1820: kmod changes
2 parents df7dd6f + 1421efa commit 0212dd1

20 files changed

+307
-62
lines changed

‎hardware_enablement/kmm-kernel-module-management.adoc‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ include::modules/kmm-replacing-in-tree-modules-with-out-of-tree-modules.adoc[lev
4141
* link:https://fastbitlab.com/building-a-linux-kernel-module/[Building a linux kernel module]
4242
4343
include::modules/kmm-example-module-cr.adoc[leveloffset=+2]
44-
include::modules/kmm-creating-moduleloader-image.adoc[leveloffset=+1]
44+
include::modules/kmm-creating-kmod-image.adoc[leveloffset=+1]
4545
include::modules/kmm-running-depmod.adoc[leveloffset=+2]
4646

4747
[role="_additional-resources"]
@@ -78,8 +78,8 @@ include::modules/kmm-using-driver-toolkit.adoc[leveloffset=+2]
7878
include::modules/kmm-using-signing-with-kmm.adoc[leveloffset=+1]
7979
include::modules/kmm-adding-the-keys-for-secureboot.adoc[leveloffset=+1]
8080
include::modules/kmm-checking-the-keys.adoc[leveloffset=+2]
81-
include::modules/kmm-signing-a-prebuilt-driver-container.adoc[leveloffset=+1]
82-
include::modules/kmm-building-and-signing-a-moduleloader-container-image.adoc[leveloffset=+1]
81+
include::modules/kmm-signing-kmods-in-a-prebuilt-image.adoc[leveloffset=+1]
82+
include::modules/kmm-building-and-signing-a-kmod-image.adoc[leveloffset=+1]
8383
[role="_additional-resources"]
8484
.Additional resources
8585
* link:https://docs.openshift.com/container-platform/4.12/authentication/understanding-and-creating-service-accounts.html#service-accounts-managing_understanding-service-accounts[Creating service accounts]
@@ -135,19 +135,21 @@ include::modules/kmm-debugging-and-troubleshooting.adoc[leveloffset=+1]
135135
// Added for TELCODOCS-1067
136136
include::modules/kmm-firmware-support.adoc[leveloffset=+1]
137137
[role="_additional-resources"]
138-
.Additional resources
139-
* xref:../hardware_enablement/kmm-kernel-module-management.adoc#kmm-creating-moduleloader-image_kernel-module-management-operator[Creating a ModuleLoader image]
140138
141139
include::modules/kmm-configuring-the-lookup-path-on-nodes.adoc[leveloffset=+2]
142140
[role="_additional-resources"]
143141
.Additional resources
144142
* xref:../post_installation_configuration/machine-configuration-tasks.adoc#understanding-the-machine-config-operator[Machine Config Operator]
145143
146-
include::modules/kmm-building-a-moduleloader-image.adoc[leveloffset=+2]
144+
include::modules/kmm-building-a-kmod-image.adoc[leveloffset=+2]
147145
include::modules/kmm-tuning-the-module-resource.adoc[leveloffset=+2]
148146
149147
// Added for TELCODOCS-1059
150148
include::modules/kmm-troubleshooting.adoc[leveloffset=+1]
149+
// Added for TELCODOCS-1820
150+
include::modules/kmm-reading-operator-logs.adoc[leveloffset=+2]
151+
include::modules/kmm-observing-events.adoc[leveloffset=+2]
152+
151153
include::modules/kmm-must-gather-tool.adoc[leveloffset=+2]
152154
[role="_additional-resources"]
153155
.Additional resources

‎modules/kmm-building-a-kmod-image.adoc‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="kmm-building-a-kmod-image_{context}"]
7+
= Building a kmod image
8+
9+
.Procedure
10+
11+
* In addition to building the kernel module itself, include the binary firmware in the builder image:
12+
+
13+
[source,dockerfile]
14+
----
15+
FROM registry.redhat.io/ubi9/ubi-minimal as builder
16+
17+
# Build the kmod
18+
19+
RUN ["mkdir", "/firmware"]
20+
RUN ["curl", "-o", "/firmware/firmware.bin", "https://artifacts.example.com/firmware.bin"]
21+
22+
FROM registry.redhat.io/ubi9/ubi-minimal
23+
24+
# Copy the kmod, install modprobe, run depmod
25+
26+
COPY --from=builder /firmware /firmware
27+
----

‎modules/kmm-building-and-signing-a-kmod-image.adoc‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="kmm-building-and-signing-a-kmod-image_{context}"]
7+
= Building and signing a kmod image
8+
9+
Use this procedure if you have source code and must build your image first.
10+
11+
The following YAML file builds a new container image using the source code from the repository. The image produced is saved back in the registry with a temporary name, and this temporary image is then signed using the parameters in the `sign` section.
12+
13+
The temporary image name is based on the final image name and is set to be `<containerImage>:<tag>-<namespace>_<module name>_kmm_unsigned`.
14+
15+
For example, using the following YAML file, Kernel Module Management (KMM) builds an image named `example.org/repository/minimal-driver:final-default_example-module_kmm_unsigned` containing the build with unsigned kmods and pushes it to the registry. Then it creates a second image named `example.org/repository/minimal-driver:final` that contains the signed kmods. It is this second image that is pulled by the worker pods and contains the kmods to be loaded on the cluster nodes.
16+
17+
After it is signed, you can safely delete the temporary image from the registry. It will be rebuilt, if needed.
18+
19+
.Prerequisites
20+
21+
* The `keySecret` and `certSecret` secrets have been created in the same namespace as the rest of the resources.
22+
23+
.Procedure
24+
25+
* Apply the YAML file:
26+
+
27+
[source,yaml]
28+
----
29+
---
30+
apiVersion: v1
31+
kind: ConfigMap
32+
metadata:
33+
name: example-module-dockerfile
34+
namespace: <namespace> <1>
35+
data:
36+
Dockerfile: |
37+
ARG DTK_AUTO
38+
ARG KERNEL_VERSION
39+
FROM ${DTK_AUTO} as builder
40+
WORKDIR /build/
41+
RUN git clone -b main --single-branch https://github.com/rh-ecosystem-edge/kernel-module-management.git
42+
WORKDIR kernel-module-management/ci/kmm-kmod/
43+
RUN make
44+
FROM registry.access.redhat.com/ubi9/ubi:latest
45+
ARG KERNEL_VERSION
46+
RUN yum -y install kmod && yum clean all
47+
RUN mkdir -p /opt/lib/modules/${KERNEL_VERSION}
48+
COPY --from=builder /build/kernel-module-management/ci/kmm-kmod/*.ko /opt/lib/modules/${KERNEL_VERSION}/
49+
RUN /usr/sbin/depmod -b /opt
50+
---
51+
apiVersion: kmm.sigs.x-k8s.io/v1beta1
52+
kind: Module
53+
metadata:
54+
name: example-module
55+
namespace: <namespace> <1>
56+
spec:
57+
moduleLoader:
58+
serviceAccountName: default <2>
59+
container:
60+
modprobe:
61+
moduleName: simple_kmod
62+
kernelMappings:
63+
- regexp: '^.*\.x86_64$'
64+
containerImage: <final_driver_container_name>
65+
build:
66+
dockerfileConfigMap:
67+
name: example-module-dockerfile
68+
sign:
69+
keySecret:
70+
name: <private_key_secret_name>
71+
certSecret:
72+
name: <certificate_secret_name>
73+
filesToSign:
74+
- /opt/lib/modules/4.18.0-348.2.1.el8_5.x86_64/kmm_ci_a.ko
75+
imageRepoSecret: <3>
76+
name: repo-pull-secret
77+
selector: # top-level selector
78+
kubernetes.io/arch: amd64
79+
----
80+
81+
<1> Replace `default` with a valid namespace.
82+
83+
<2> The default `serviceAccountName` does not have the required permissions to run a module that is privileged. For information on creating a service account, see "Creating service accounts" in the "Additional resources" of this section.
84+
85+
<3> Used as `imagePullSecrets` in the `DaemonSet` object and to pull and push for the build and sign features.

‎modules/kmm-building-in-cluster.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
= Building in the cluster
99

10-
KMM can build module loader images in the cluster. Follow these guidelines:
10+
KMM can build kmod images in the cluster. Follow these guidelines:
1111

1212
* Provide build instructions using the `build` section of a kernel mapping.
1313
* Copy the `Dockerfile` for your container image into a `ConfigMap` resource, under the `dockerfile` key.

‎modules/kmm-creating-kmod-image.adoc‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * hardware_enablement/kmm-kernel-module-management.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="kmm-creating-kmod-image_{context}"]
7+
= Creating a kmod image
8+
9+
Kernel Module Management (KMM) works with purpose-built kmod images, which are standard OCI images that contain `.ko` files.
10+
The location of the `.ko` files must match the following pattern: `<prefix>/lib/modules/[kernel-version]/`.
11+
12+
Keep the following in mind when working with the `.ko` files:
13+
14+
* In most cases, `<prefix>` should be equal to `/opt`. This is the `Module` CRD's default value.
15+
* `kernel-version` must not be empty and must be equal to the kernel version the kernel modules were built for.

‎modules/kmm-creating-module-cr.adoc‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,21 +7,20 @@
77

88
= The Module custom resource definition
99

10-
The `Module` custom resource definition (CRD) represents a kernel module that can be loaded on all or select nodes in the cluster, through a module loader image.
11-
A `Module` custom resource (CR) specifies one or more kernel versions with which it is compatible, and a node selector.
10+
The `Module` custom resource definition (CRD) represents a kernel module that can be loaded on all or select nodes in the cluster, through a kmod image. A `Module` custom resource (CR) specifies one or more kernel versions with which it is compatible, and a node selector.
1211

13-
The compatible versions for a `Module` resource are listed under `.spec.moduleLoader.container.kernelMappings`.
14-
A kernel mapping can either match a `literal` version, or use `regexp` to match many of them at the same time.
12+
The compatible versions for a `Module` resource are listed under `.spec.moduleLoader.container.kernelMappings`. A kernel mapping can either match a `literal` version, or use `regexp` to match many of them at the same time.
1513

1614
The reconciliation loop for the `Module` resource runs the following steps:
1715

1816
. List all nodes matching `.spec.selector`.
1917
. Build a set of all kernel versions running on those nodes.
2018
. For each kernel version:
21-
.. Go through `.spec.moduleLoader.container.kernelMappings` and find the appropriate container image name. If the kernel mapping has `build` or `sign` defined and the container image does not already exist, run the build, the signing job, or both, as needed.
22-
.. Create a module loader daemon set with the container image determined in the previous step.
23-
.. If `.spec.devicePlugin` is defined, create a device plugin daemon set using the configuration specified under `.spec.devicePlugin.container`.
19+
.. Go through `.spec.moduleLoader.container.kernelMappings` and find the appropriate container image name.
20+
If the kernel mapping has `build` or `sign` defined and the container image does not already exist, run the build, the signing pod, or both, as needed.
21+
.. Create a worker pod to pull the container image determined in the previous step and run `modprobe`.
22+
.. If `.spec.devicePlugin` is defined, create a device plugin daemon set using the configuration specified under `.spec.devicePlugin.container`.
2423
. Run `garbage-collect` on:
25-
.. Existing daemon set resources targeting kernel versions that are not run by any node in the cluster.
26-
.. Successful build jobs.
27-
.. Successful signing jobs.
24+
.. Obsolete device plugin `DaemonSets` that do not target any node.
25+
.. Successful build pods.
26+
.. Successful signing pods.

‎modules/kmm-creating-moduleloader-image.adoc‎

Lines changed: 0 additions & 13 deletions
This file was deleted.

‎modules/kmm-deploying-modules.adoc‎

Lines changed: 10 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,18 @@
66
[id="kmm-deploy-kernel-modules_{context}"]
77
= Kernel module deployment
88

9-
For each `Module` resource, Kernel Module Management (KMM) can create a number of `DaemonSet` resources:
9+
Kernel Module Management (KMM) monitors `Node` and `Module` resources in the cluster to determine if a kernel module should be loaded on or unloaded from a node.
1010

11-
* One ModuleLoader `DaemonSet` per compatible kernel version running in the cluster.
12-
* One device plugin `DaemonSet`, if configured.
11+
To be eligible for a module, a node must contain the following:
1312

14-
The module loader daemon set resources run ModuleLoader images to load kernel modules.
15-
A module loader image is an OCI image that contains the `.ko` files and both the `modprobe` and `sleep` binaries.
13+
* Labels that match the module's `.spec.selector` field.
14+
* A kernel version matching one of the items in the module's `.spec.moduleLoader.container.kernelMappings` field.
15+
* If ordered upgrade (`ordered_upgrade.md`) is configured in the module, a label that matches its `.spec.moduleLoader.container.version` field.
1616
17-
When the module loader pod is created, the pod runs `modprobe` to insert the specified module into the kernel.
18-
It then enters a sleep state until it is terminated.
19-
When that happens, the `ExecPreStop` hook runs `modprobe -r` to unload the kernel module.
17+
When KMM reconciles nodes with the desired state as configured in the `Module` resource, it creates worker pods on the target nodes to run the necessary action. The KMM Operator monitors the outcome of the pods and records the information. The Operator uses this information to label the `Node` objects when the module is successfully loaded, and to run the device plugin, if configured.
2018

21-
If the `.spec.devicePlugin` attribute is configured in a `Module` resource, then KMM creates a link:https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/[device plugin]
22-
daemon set in the cluster.
23-
That daemon set targets:
19+
Worker pods run the KMM `worker` binary that performs the following tasks:
2420

25-
* Nodes that match the `.spec.selector` of the `Module` resource.
26-
* Nodes with the kernel module loaded (where the module loader pod is in the `Ready` condition).
21+
* Pulls the kmod image configured in the `Module` resource. Kmod images are standard OCI images that contain `.ko` files.
22+
* Extracts the image in the pod's filesystem.
23+
* Runs `modprobe` with the specified arguments to perform the necessary action.

‎modules/kmm-firmware-support.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
[id="kmm-firmware-support_{context}"]
77
= KMM firmware support
88

9-
Kernel modules sometimes need to load firmware files from the file system. KMM supports copying firmware files from the ModuleLoader image to the node's file system.
9+
Kernel modules sometimes need to load firmware files from the file system. KMM supports copying firmware files from the kmod image to the node's file system.
1010

1111
The contents of `.spec.moduleLoader.container.modprobe.firmwarePath` are copied into the `/var/lib/firmware` path on the node before running the `modprobe` command to insert the kernel module.
1212

‎modules/kmm-gathering-data-for-kmm-hub.adoc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@
1414
+
1515
[source,terminal]
1616
----
17-
$ export MUST_GATHER_IMAGE=$(oc get deployment -n openshift-kmm-hub kmm-operator-hub-controller -ojsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env[?(@.name=="RELATED_IMAGES_MUST_GATHER")].value}')
17+
$ export MUST_GATHER_IMAGE=$(oc get deployment -n openshift-kmm-hub kmm-operator-hub-controller -ojsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env[?(@.name=="RELATED_IMAGE_MUST_GATHER")].value}')
18+
$ oc adm must-gather --image="${MUST_GATHER_IMAGE}" -- /usr/bin/gather -u
1819
----
1920
+
2021
[NOTE]

0 commit comments

Comments
 (0)