Skip to content

Commit d8b38aa

Browse files
ShaunaDiazShaunaDiaz
authored
Merge pull request #97090 from tedaveryredhat/OSDOCS-13847
OSDOCS-13847:Generic Device Plugin Feature
2 parents ff0da25 + 86cf18f commit d8b38aa

8 files changed

+385
-0
lines changed

‎_topic_maps/_topic_map_ms.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,8 @@ Topics:
141141
File: microshift-low-latency
142142
- Name: Workload partitioning
143143
File: microshift-workload-partitioning
144+
- Name: About the Generic Device Plugin (GDP)
145+
File: microshift-gdp
144146
---
145147
Name: Networking
146148
Dir: microshift_networking

‎microshift_configuring/microshift-gdp.adoc‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="microshift-gdp"]
3+
= Using the Generic Device Plugin
4+
include::_attributes/attributes-microshift.adoc[]
5+
:context: microshift-generic-device-plugin
6+
7+
:FeatureName: The Generic Device Plugin for {microshift-short}
8+
9+
include::snippets/technology-preview.adoc[]
10+
11+
toc::[]
12+
13+
The Generic Device Plugin (GDP) for {microshift-short} enables your containerized applications to securely access physical host devices, such as serial ports, video cameras, or sound cards directly from within Kubernetes pods. By using GDP, you can extend capabilities of {microshift-short} to support applications that require direct hardware interaction, such as Internet of Things (IoT) applications.
14+
15+
include::modules/microshift-con-understanding-generic-device-plugin.adoc[leveloffset=+1]
16+
17+
include::modules/microshift-con-limitations-of-generic-device-plugin.adoc[leveloffset=+1]
18+
19+
include::modules/microshift-proc-configuring-generic-device-plugin.adoc[leveloffset=+1]
20+
21+
include::modules/microshift-proc-deploying-applications-with-generic-devices.adoc[leveloffset=+1]
22+
23+
include::modules/microshift-ref-generic-device-plugin-configuration-parameters.adoc[leveloffset=+1]
24+
25+
include::modules/microshift-ref-generic-device-plugin-troubleshooting.adoc[leveloffset=+1]
26+
27+
[id="_additional-resources_microshift-gdp_{context}"]
28+
== Additional resources
29+
30+
* xref:../microshift_configuring/microshift-using-config-yaml.adoc#microshift-config-yaml_microshift-configuring[The {microshift-short} YAML configuration file]
31+
* xref:../microshift_networking/microshift-networking-settings.adoc#microshift-understanding-networking-settings[Understanding networking settings]
32+
* xref:../microshift_networking/microshift_multiple_networks/microshift-cni-multus.adoc#microshift-cni-multus[About using multiple networks]
33+
* xref:../microshift_networking/microshift_network_policy/microshift-network-policy-index.adoc#microshift-network-policies[About network policies]
34+

‎modules/microshift-con-limitations-of-generic-device-plugin.adoc‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_configuring/microshift-gdp.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="microshift-limitations-of-generic-device-plugin_{context}"]
7+
= Limitations and considerations for the Generic Device Plugin
8+
9+
Although the Generic Device Plugin (GDP) provides powerful capabilities for accessing host devices in {microshift-short}, it is important to understand its limitations and current status.
10+
11+
[id="microshift-devices-not-suited-generic-device-plugin_{context}"]
12+
== Devices not suited for the Generic Device Plugin
13+
14+
The GDP is designed for devices that are managed directly by the operating system and do not require special setup procedures. Devices that are not well-suited for the Generic Device Plugin include:
15+
16+
* Complex hardware requiring specialized drivers such as GPUs (graphics processing units) or FPGAs (field-programmable gate arrays). These types of hardware typically require dedicated device plugins that can perform unique initialization procedures, memory management, or queue resets before a pod can use them.
17+
* Devices with specific vendor-supplied software stacks. Devices that require a complex software stack or proprietary APIs beyond direct file system access might require a specialized plugin.
18+
19+
[id="microshift-device-id-logging-generic-device-plugin_{context}"]
20+
== Device identification and logging
21+
22+
When you use glob paths, for example, `/dev/ttyUSB*`, to expose multiple similar devices, the GDP allocates devices based on availability. However, if your application needs to connect to an exactly specified physical device, for example, `serial device 3` out of 10, using broad glob paths might be insufficient. In such cases, configure individual device entries in the `config.yaml` file using more stable and unique identifiers such as:
23+
24+
* Specific device paths, for example, `/dev/video0`.
25+
* Symbolic links provided by the operating system, for example, `/dev/serial/by-id/` or `/dev/serial/by-path/`.
26+
* USB vendor ID, product ID, and serial number combinations for precise USB device targeting.
27+
28+
[id="microshift-performance-considerations-generic-device-plugin_{context}"]
29+
== Performance considerations
30+
31+
The `count` parameter in the `config.yaml` file enables a device group to be scheduled multiple times concurrently. While there are no explicit limits set within the GDP for the count (for example, `1000` for `/dev/fuse`), the actual performance depends on the host system's capabilities and the nature of the device itself. Running a very high number of concurrent processes that access the same device might affect performance.

‎modules/microshift-con-understanding-generic-device-plugin.adoc‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_configuring/microshift-gdp.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="microshift-understanding-generic-device-plugin-con_{context}"]
7+
= Understanding the Generic Device Plugin
8+
9+
The Generic Device Plugin (GDP) is a Kubernetes device plugin that enables applications running in pods to access host devices such as serial ports, cameras, and sound cards securely. This capability is especially important for edge and IoT environments where direct hardware interaction is a common requirement. The GDP integrates with the kubelet to advertise available devices to the cluster and facilitate their allocation to pods without requiring elevated privileges within the container itself.
10+
11+
The GDP is designed to handle devices that are initialized and managed by the operating system and do not require any special initialization procedures or drivers for a pod to use them.
12+
13+
Here are examples of generic devices that are suitable for the GDP:
14+
15+
* Serial ports, for example, `/dev/ttyUSB*`, `/dev/ttyACM*`.
16+
* Video cameras, for example, `/dev/video0`.
17+
* Sound devices, for example, `/dev/snd`, `/dev/snd/controlC0`.
18+
* USB devices specified by Vendor ID and Product ID, or, optionally, by the device serial number.
19+
20+
The following specialized devices are not suitable for the GDP:
21+
22+
* Devices that require specific initialization procedures beyond standard operating system management.
23+
* Specialized hardware that needs additional drivers or kernel modules. Examples of this specialized hardware include GPUs and FPGAs. These types of devices typically require their own specialized device plugins.

‎modules/microshift-proc-configuring-generic-device-plugin.adoc‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_configuring/microshift-gdp.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-configuring-generic-device-plugin_{context}"]
7+
= Configuring the Generic Device Plugin
8+
9+
:FeatureName: The Generic Device Plugin for {microshift-short}
10+
11+
include::snippets/technology-preview.adoc[]
12+
13+
The Generic Device Plugin (GDP) is disabled by default in {microshift-short}. To use the GDP, you must enable it and specify which host devices your Kubernetes applications can access. To enable the GDP, you must modify the {microshift-short} `config.yaml` file or create a configuration snippet file. For example, `/etc/microshift/config.d/10-gdp.yaml`.
14+
15+
.Prerequisites
16+
17+
* You installed {microshift-short}.
18+
* You created a custom `config.yaml` file in the `/etc/microshift` directory.
19+
* You installed the {oc-first}.
20+
* You have `sudo` privileges on the {microshift-short} host.
21+
* You have identified the specific host devices that you want to expose to your {microshift-short} cluster. For example, `/dev/video0`, `/dev/ttyUSB*`, or USB Vendor/Product IDs.
22+
23+
.Procedure
24+
25+
. From your CLI using `sudo` privileges, open `/etc/microshift/config.yaml` in a text editor.
26+
+
27+
. Locate the `genericDevicePlugin` section. If it is not present, add it.
28+
+
29+
. Set the `status` parameter to `Enabled` and define the `devices` that should be exposed. Each device definition needs a `name` and one or more `groups`. Each group can specify devices using `paths`, for file-based devices, including glob patterns, or `usbs`, for USB devices using Vendor/Product IDs. You cannot mix `paths` and `usbs` within the same device group.
30+
+
31+
.GDP fields with default values
32+
[source,yaml]
33+
----
34+
apiServer:
35+
# ...
36+
genericDevicePlugin:
37+
devices:
38+
- groups:
39+
- paths:
40+
- path: /dev/ttyUSB* <1>
41+
- path: /dev/ttyACM* <2>
42+
name: serial
43+
- groups:
44+
- paths:
45+
- path: /dev/fuse <3>
46+
name: fuse <4>
47+
- groups:
48+
- usbs: <5>
49+
- product: "0x7523" <6>
50+
serial: ""
51+
vendor: "0x1a86" <7>
52+
name: converter
53+
domain: device.microshift.io <8>
54+
status: Enabled
55+
----
56+
<1> Exposes all the USB serial devices that are matched by this glob.
57+
<2> Exposes all the ACM serial devices that are matched by this glob.
58+
<3> For example, the file path for a fuse device.
59+
<4> For example, the name of the device.
60+
<5> Exposes a specific USB device by Vendor ID and Product ID.
61+
<6> For example, the Product ID for a CH340 serial converter
62+
<7> For example, the Vendor ID for a CH340 serial converter.
63+
<8> Default domain for the GDP.
64+
+
65+
[IMPORTANT]
66+
====
67+
* The output of the `microshift show-config` parameter might include pre-configured default paths for serial devices even if you have not explicitly configured them in `config.yaml`. These paths represent the default discovery settings if the Generic Device Plugin is enabled without specific user configuration.
68+
69+
* For consistency and precise device targeting, especially when dealing with multiple similar devices, consider using stable device paths like `/dev/serial/by-id/` or specific USB Vendor, Product, or Serial IDs instead of broad glob patterns like `/dev/ttyUSB*`.
70+
71+
* The `count` parameter in a device group allows a single device, or a set of devices matched by a glob, to be allocated multiple times concurrently to different pods. If omitted, `count` defaults to `1`.
72+
====
73+
+
74+
. Save the `config.yaml` file.
75+
76+
. Restart the {microshift-short} service to apply the changes:
77+
+
78+
[source,terminal]
79+
----
80+
$ sudo systemctl restart microshift
81+
----
82+
+
83+
Allow some time for {microshift-short} to restart and for the GDP to register its devices with the Kubelet.
84+
85+
.Verification
86+
87+
* You can check the available devices in your cluster by running the following command:
88+
+
89+
[source,terminal,subs="+quotes"]
90+
----
91+
$ oc describe node _<microshift_node_name>_ | grep "device.microshift.io" <1>
92+
----
93+
<1> Replace _<microshift_node_name>_ with your node name.
94+
+
95+
* Depending on your configuration, expect output that indicates that the devices are now discoverable and schedulable within your {microshift-short} cluster.
96+
+
97+
.Example output
98+
[source,terminal]
99+
----
100+
Capacity:
101+
cpu: 2
102+
device.microshift.io/audio: 0
103+
device.microshift.io/capture: 0
104+
device.microshift.io/custom-device: 1
105+
device.microshift.io/dummy-video: 0
106+
device.microshift.io/fuse: 0
107+
device.microshift.io/serial: 5
108+
device.microshift.io/video: 0
109+
Allocatable:
110+
cpu: 2
111+
device.microshift.io/audio: 0
112+
device.microshift.io/capture: 0
113+
device.microshift.io/custom-device: 1
114+
device.microshift.io/dummy-video: 0
115+
device.microshift.io/fuse: 0
116+
device.microshift.io/serial: 5
117+
device.microshift.io/video: 0
118+
Allocated resources:
119+
(Total limits may be over 100 percent, i.e., overcommitted.)
120+
Resource Requests Limits
121+
-------- -------- ------
122+
cpu 450m (22%) 500m (25%)
123+
memory 1550Mi (42%) 500Mi (13%)
124+
ephemeral-storage 0 (0%) 0 (0%)
125+
hugepages-1Gi 0 (0%) 0 (0%)
126+
hugepages-2Mi 0 (0%) 0 (0%)
127+
device.microshift.io/audio 0 0
128+
device.microshift.io/capture 0 0
129+
device.microshift.io/custom-device 1 1
130+
device.microshift.io/dummy-video 1 1
131+
device.microshift.io/fuse 0 0
132+
device.microshift.io/serial 0 0
133+
device.microshift.io/video 0 0
134+
----

‎modules/microshift-proc-deploying-applications-with-generic-devices.adoc‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_configuring/microshift-gdp.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-deploying-applications-with-generic-devices_{context}"]
7+
= Deploying applications that use generic devices
8+
9+
After the Generic Device Plugin (GDP) is configured and enabled in {microshift-short}, you can deploy Kubernetes workloads, such as pods, deployments, or `StatefulSets`, that request access to the host devices that you have exposed. Devices are made available inside the container without requiring the pod to run with elevated privileges.
10+
11+
.Prerequisites
12+
13+
* You installed {microshift-short}.
14+
* You enabled and configured GDP.
15+
* You installed {oc-first}.
16+
17+
.Procedure
18+
19+
. Define the device request in your `Pod` specification:
20+
+
21+
[source,yaml]
22+
----
23+
apiVersion: v1
24+
kind: Pod
25+
metadata:
26+
name: device-app
27+
spec:
28+
containers:
29+
- name: container
30+
image: <your_application_image> <1>
31+
command: ["/path/to/your/app"] <2>
32+
args: ["--device_path=/dev/video0"] <3>
33+
resources:
34+
limits: <4>
35+
device.microshift.io/video: 1 <5>
36+
securityContext: <6>
37+
allowPrivilegeEscalation: false
38+
capabilities:
39+
drop: ["ALL"]
40+
runAsNonRoot: true
41+
seccompProfile:
42+
type: "RuntimeDefault"
43+
----
44+
<1> Replace with your container image.
45+
<2> Replace with the command for your application.
46+
<3> For example, how your application might use the device.
47+
<4> The resource name must follow the pattern `device.microshift.io/<device_name>`, where `<device_name>` matches the `name` that you specified in your configuration file.
48+
<5> A request for one instance of the `video` device.
49+
<6> Define and configure with the least privilege value to ensure that the container has only required permissions, such as access to the device file, and to restrict other capabilities for the container.
50+
+
51+
. Deploy the Kubernetes workload by applying the manifest to the {microshift-short} cluster by running the following command:
52+
+
53+
[source,terminal,subs="+quotes"]
54+
----
55+
$ oc apply -f _<your-workload-manifest.yaml>_ <1>
56+
----
57+
<1> Replace _<your_workload_manifest>_ with the name of your workload manifest.
58+
59+
.Results
60+
61+
After the pod is running, the specified host device is available at its original path, or `mountPath` if specified, inside the container. Your application can then interact with it as if it were a local device.
62+
63+
For example, if you requested `device.microshift.io/serial`, which maps to `/dev/ttyUSB*`, your application might find the device at `/dev/ttyUSB0` or a similar path inside the container.
64+
65+
.Verification
66+
67+
* Verify device access by running the following command inside the running pod:
68+
+
69+
[source,terminal,subs="+quotes"]
70+
----
71+
$ oc exec -it _<pod_name>_ -- ls -l /dev/video0 <1>
72+
----
73+
<1> Replace _<pod_name>_ with the name of the pod.
74+
+

‎modules/microshift-ref-generic-device-plugin-configuration-parameters.adoc‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
// Module included in the following assemblies:
2+
//
3+
// microshift_configuring/microshift-gdp.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="microshift-generic-device-plugin-configuration-parameters_{context}"]
7+
= Generic Device Plugin configuration reference
8+
9+
This section provides a detailed reference for the configuration parameters available for the Generic Device Plugin within the {microshift-short} `config.yaml` file.
10+
11+
.GDP configuration fields definitions table
12+
[cols="1,2"]
13+
|===
14+
|Parameter|Description
15+
16+
|`genericDevicePlugin`
17+
|The `genericDevicePlugin` section of the {microshift-short} configuration file defines the configurable parameters for the implementation of the {microshift-short} `GenericDevicePlugin` API. All of the following parameters in this table are subsections in the `genericDevicePlugin` section of the {microshift-short} configuration.
18+
19+
|`devices`
20+
|Is a subgroup that lists the device definitions to be exposed by the plugin. Each `Device` entry contains a 'name' and a list of `groups`.
21+
22+
|`devices.groups`
23+
|Lists device groups. Devices within a group comprise a pool of devices under a common name. When you request a device from that pool, you can receive a device from different defined paths.
24+
25+
|`devices.groups.count`
26+
|Specifies how many times this group of devices can be mounted concurrently. If unspecified, `Count` defaults to `1`. Setting a high count, for example, `1000` for `/dev/fuse`, is possible because are no inherent limits, but performance might be impacted depending on the host's capabilities and the nature of the device.
27+
28+
|`devices.groups.paths`
29+
|Lists the host device file paths. Paths can be glob patterns, for example, `/dev/ttyUSB*`, in which case each matched device is schedulable `Count` times. This field is exclusive with `usbs`; you cannot define both in the same device group.
30+
31+
|`devices.groups.paths.limit`
32+
|Specifies up to how many times this device can be used in the group concurrently when other devices in the group yield more matches. For example, if one path in the group matches 5 devices and another matches 1 device but has a limit of 10, then the group provides 5 pairs of devices. When unspecified, the limit defaults to `1`.
33+
34+
|`devices.groups.paths.mountPath`
35+
|The file path at which the host device should be mounted within the container. When unspecified, `mountPath` defaults to `path`.
36+
37+
|`devices.groups.paths.path`
38+
|The file path of a device on the host, for example, `/dev/video0`, `/dev/ttyUSB*`.
39+
40+
|`devices.groups.paths.permissions`
41+
|The file-system permissions given to the mounted device. Applies only to mounts of type `Device`. Can be one or more of:
42+
43+
* `r` - allows the container to read from the specified device.
44+
45+
* `w` - allows the container to write to the specified device.
46+
47+
* `m` - allows the container to create device files that do not yet exist.
48+
49+
When unspecified, `Permissions` defaults to `mrw`.
50+
51+
|`devices.groups.paths.readOnly`
52+
|Specifies whether the path should be mounted read-only. The values are `true` or `false`. Applies only to mounts of type `Mount`.
53+
54+
|`devices.groups.path.type`
55+
|Describes what type of file-system node this `path` represents and thus how it should be mounted. The type can be `Device` or `Mount`. When unspecified, `type` defaults to `Device`.
56+
57+
|`devices.groups.usbs`
58+
|Lists the USB specifications that this device group consists of. The vendor and product IDs must always match. The serial ID must match if provided, or skipped if the ID is empty, The `usbs` field is exclusive with `paths`.
59+
60+
|`devices.groups.usbs.product`
61+
|The USB Product ID of the device to match on, for example, `0x7523`.
62+
63+
|`devices.groups.usbs.serial`
64+
|The serial number of the device to match on. A USB device must match exactly on all the given attributes to pass.
65+
66+
|`devices.groups.usbs.vendor`
67+
|The USB Vendor ID of the device to match on, for example, `0x1a86`
68+
69+
|`devices.name`
70+
|A unique string representing the kind of device this specification describes, for example, `serial`, `video`, or `fuse`. This name is used in pod resource requests, for example, `device.microshift.io/serial`.
71+
72+
|`domain`
73+
|`domain` is a subgroup that specifies the domain prefix with which devices are advertised and present in the cluster. For example, `device.microshift.io/serial`. The default value is `device.microshift.io`.
74+
75+
|`status`
76+
|`status` is a subgroup that specifies the default GDP status. `Enabled` or `Disabled` are valid values.
77+
78+
|===

0 commit comments

Comments
 (0)