Skip to content

Commit fcc22f9

Browse files
JoeAldingerJoeAldinger
authored
Merge pull request #103509 from StephenJamesSmith/OSDOCS-17092
OSDOCS-17092: Enable monitoring of pending workloads
2 parents 3f49edc + 52ca2ea commit fcc22f9

8 files changed

+438
-0
lines changed

‎_topic_maps/_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3457,6 +3457,8 @@ Topics:
34573457
File: configuring-quotas
34583458
- Name: Managing jobs and workloads
34593459
File: managing-workloads
3460+
- Name: Monitoring pending workloads
3461+
File: monitoring-pending-workloads
34603462
- Name: Using cohorts
34613463
File: using-cohorts
34623464
- Name: Configuring fair sharing

‎ai_workloads/kueue/monitoring-pending-workloads.adoc‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
[id="monitoring-pending-workloads-install-kueue"]
4+
= Monitoring pending workloads
5+
:context: monitoring-pending-workloads
6+
7+
toc::[]
8+
9+
[role="_abstract"]
10+
{kueue-name} provides the `VisibilityOnDemand` feature to monitor pending workloads. A workload is an application that runs to completion. It can be composed by one or multiple pods that, loosely or tightly coupled, as a whole, complete a task. A workload is the unit of admission in {kueue-name}.
11+
12+
The `VisibilityOnDemand` feature provides the ability for batch administrators to monitor the pipeline of pending jobs in the cluster queue and the local queue and batch users just for local queue, and help users to estimate when their jobs will start.
13+
14+
You can regulate inbound requests and high request volumes, and provide user permissions for viewing the pending workloads.
15+
16+
include::modules/kueue-configuring-api-priority-and-fairness.adoc[leveloffset=+1]
17+
18+
[role="_additional-resources"]
19+
.Additional resources
20+
21+
* link:https://kubernetes.io/docs/concepts/cluster-administration/flow-control/[API Priority and Fairness]
22+
23+
include::modules/kueue-providing-user-permissions.adoc[leveloffset=+1]
24+
25+
[role="_additional-resources"]
26+
.Additional resources
27+
28+
* link:https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue#rbac-permissions[Configuring role-based permissions]
29+
30+
31+
include::modules/kueue-monitoring-pending-workloads-on-demand.adoc[leveloffset=+1]
32+
33+
include::modules/kueue-viewing-pending-workloads-clusterqueue.adoc[leveloffset=+2]
34+
35+
include::modules/kueue-viewing-pending-workloads-localqueue.adoc[leveloffset=+2]
36+
37+
include::modules/kueue-modifying-monitoring-settings.adoc[leveloffset=+1]
38+

‎modules/kueue-configuring-api-priority-and-fairness.adoc‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * ai_workloads/kueue/monitoring-pending-workloads.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="configuring-api-priority-and-fairness_{context}"]
7+
= API Priority and Fairness
8+
9+
[role="_abstract"]
10+
{kueue-name} uses Kubernetes API Priority and Fairness (APF) To help manage pending workloads. APF is a flow control mechanism that allows you to define API-level policies to regulate inbound requests to the API server. It protects the API server from being overwhelmed by unexpectedly high request volume, while protecting critical traffic from the throttling effect on best-effort workloads.
11+
12+
13+

‎modules/kueue-modifying-monitoring-settings.adoc‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * ai_workloads/kueue/monitoring-pending-workloads.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="modifying-monitoring-settings_{context}"]
7+
= Modifying monitoring settings
8+
9+
[role="_abstract"]
10+
Modify the monitoring settings according to your organization's requirements to ensure users can access and view the pending workloads in a timely and reliable manner.
11+
12+
This procedure tells you how to modify the resource flow control for the {kueue-name} `VisibilityOnDemand` feature. Modifications directly impact the system's ability to handle concurrent requests for job visibility information.
13+
14+
.Procedure
15+
. Edit the `PriorityLevelConfiguration` asset for `VisibilityOnDemand` on `Kueue` by running the following command:
16+
+
17+
[source,terminal]
18+
----
19+
$ oc edit prioritylevelconfiguration kueue-visibility
20+
----
21+
22+
. Modify the `nominalConcurrencyShares` field in the `PriorityLevelConfiguration` asset by setting the value for `kueue.openshift.io/allow-nominal-concurrency-shares-update` to `true`.
23+
+
24+
The possible values you can specify for `nominalConcurrencyShares` are `0`, `2` (the default) until `5`. If you specify a value that is not acceptable (the value `1` or any value above `5`), the default value `2`, is enforced.
25+
+
26+
See the following example:
27+
+
28+
[source,yaml]
29+
----
30+
apiVersion: flowcontrol.apiserver.k8s.io/v1
31+
kind: PriorityLevelConfiguration
32+
metadata:
33+
name: kueue-visibility
34+
annotations:
35+
kueue.openshift.io/allow-nominal-concurrency-shares-update: "false"
36+
spec:
37+
limited:
38+
borrowingLimitPercent: 0
39+
lendablePercent: 90
40+
limitResponse:
41+
queuing:
42+
handSize: 4
43+
queueLengthLimit: 50
44+
queues: 16
45+
type: Queue
46+
nominalConcurrencyShares: 2
47+
type: Limited
48+
----
49+
+
50+
The default value for `kueue.openshift.io/allow-nominal-concurrency-shares-update` is `false`. If you change the value of `nominalConcurrencyShares` to any value other than `2`, then you must first change the value of `kueue.openshift.io/allow-nominal-concurrency-shares-update` to `true`. Otherwise, the value you assign for `nominalConcurrencyShares` will not take effect.
51+
52+
. Verify the value is kept by running the following command:
53+
+
54+
[source,terminal]
55+
----
56+
$ oc get prioritylevelconfiguration kueue-visibility
57+
----

‎modules/kueue-monitoring-pending-workloads-on-demand.adoc‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * ai_workloads/kueue/monitoring-pending-workloads.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="monitoring-pending-workloads-on-demand_{context}"]
7+
= Monitoring pending workloads on demand
8+
9+
[role="_abstract"]
10+
To test the monitoring of pending workloads, you must correctly configure both the `ClusterQueue` and the `LocalQueue` resources. After that, you can create jobs on that `LocalQueue`. Kueue manages the workload object created from the job so, when a job is submitted and saturates the `ClusterQueue`, its corresponding workloads can be seen in the list of pending workloads.
11+
12+
.Prerequisites
13+
14+
* You have cluster administrator permissions.
15+
* The {kueue-name} Operator is installed on your cluster, and you have created a `Kueue` custom resource (CR).
16+
* You have installed the {oc-first}.
17+
* The {oc-first} has communication with your cluster.
18+
19+
The following procedure tells you how to install and test workload monitoring.
20+
21+
.Procedure
22+
23+
. Create the assets by running the following command:
24+
+
25+
[source,terminal]
26+
----
27+
cat <<EOF| oc create -f -
28+
---
29+
apiVersion: kueue.x-k8s.io/v1beta1
30+
kind: ResourceFlavor
31+
metadata:
32+
name: "default-flavor"
33+
---
34+
apiVersion: kueue.x-k8s.io/v1beta1
35+
kind: ClusterQueue
36+
metadata:
37+
name: "cluster-queue"
38+
spec:
39+
namespaceSelector: {} # match all.
40+
resourceGroups:
41+
- coveredResources: ["cpu", "memory"]
42+
flavors:
43+
- name: "default-flavor"
44+
resources:
45+
- name: "cpu"
46+
nominalQuota: 9
47+
- name: "memory"
48+
nominalQuota: 36Gi
49+
---
50+
apiVersion: kueue.x-k8s.io/v1beta1
51+
kind: LocalQueue
52+
metadata:
53+
namespace: "default"
54+
name: "user-queue"
55+
spec:
56+
clusterQueue: "cluster-queue"
57+
---
58+
EOF
59+
----
60+
61+
. Create the following file with the job manifest:
62+
+
63+
[source,terminal]
64+
----
65+
cat >> job.yaml << EOF
66+
apiVersion: batch/v1
67+
kind: Job
68+
metadata:
69+
generateName: sample-job-
70+
namespace: default
71+
labels:
72+
kueue.x-k8s.io/queue-name: user-queue
73+
spec:
74+
parallelism: 3
75+
completions: 3
76+
suspend: true
77+
template:
78+
spec:
79+
containers:
80+
- name: <example-job>
81+
image: registry.k8s.io/e2e-test-images/agnhost:2.53
82+
command: [ "/bin/sh" ]
83+
args: [ "-c", "sleep 60" ]
84+
resources:
85+
requests:
86+
cpu: "1"
87+
memory: "200Mi"
88+
restartPolicy: Never
89+
EOF
90+
----
91+
92+
. Label the default namespace to be managed by Kueue by running the following command:
93+
+
94+
[source,terminal]
95+
----
96+
$ oc label namespace default kueue.openshift.io/managed=true
97+
----
98+
99+
. Create the six jobs by running the following command:
100+
+
101+
[source,terminal]
102+
----
103+
for i in {1..6}; do oc create -f job.yaml; done
104+
----
105+
+
106+
In this example, three of the jobs saturate the `ClusterQueue` resource and the other three jobs should be pending.

‎modules/kueue-providing-user-permissions.adoc‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * ai_workloads/kueue/monitoring-pending-workloads.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="providing-user-permissions_{context}"]
7+
= Providing user permissions
8+
9+
[role="_abstract"]
10+
You can configure role-based access control (RBAC) objects for the users of your {kueue-name} deployment. These objects determine which types of users can create which types of {kueue-name} objects.
11+
12+
You need to provide permissions to the users that require access to the specific APIs.
13+
14+
* If the user needs access to the pending workloads from the `ClusterQueue` resource, a `ClusterRoleBinding` schema needs to be created referencing the ClusterRole `kueue-batch-admin-role`.
15+
16+
* If the user needs access to the pending workloads from the `LocalQueue` resource, a `RoleBinding` schema needs to be created referencing the ClusterRole `kueue-batch-user-role`.

‎modules/kueue-viewing-pending-workloads-clusterqueue.adoc‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * ai_workloads/kueue/monitoring-pending-workloads.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="viewing-pending-workloads-clusterqueue_{context}"]
7+
= Viewing pending workloads in ClusterQueue
8+
9+
[role="_abstract"]
10+
To view all pending workloads at the cluster level, administrators can use the `ClusterQueue` object visibility endpoint of Kueue's visibility API. This endpoint returns a list of all workloads currently waiting for admission by that `ClusterQueue` resource.
11+
12+
.Procedure
13+
14+
. To view pending workloads in `ClusterQueue` run the following command:
15+
+
16+
[source,terminal]
17+
----
18+
$ oc get --raw "/apis/visibility.kueue.x-k8s.io/v1beta1/clusterqueues/cluster-queue/pendingworkloads"
19+
----
20+
+
21+
.Example output
22+
[source,yaml]
23+
----
24+
{
25+
"kind": "PendingWorkloadsSummary",
26+
"apiVersion": "visibility.kueue.x-k8s.io/v1beta1",
27+
"metadata": {
28+
"creationTimestamp": null
29+
},
30+
"items": [
31+
{
32+
"metadata": {
33+
"name": "job-sample-job-jrjfr-8d56e",
34+
"namespace": "default",
35+
"creationTimestamp": "2023-12-05T15:42:03Z",
36+
"ownerReferences": [
37+
{
38+
"apiVersion": "batch/v1",
39+
"kind": "Job",
40+
"name": "sample-job-jrjfr",
41+
"uid": "5863cf0e-b0e7-43bf-a445-f41fa1abedfa"
42+
}
43+
]
44+
},
45+
"priority": 0,
46+
"localQueueName": "user-queue",
47+
"positionInClusterQueue": 0,
48+
"positionInLocalQueue": 0
49+
},
50+
{
51+
"metadata": {
52+
"name": "job-sample-job-jg9dw-5f1a3",
53+
"namespace": "default",
54+
"creationTimestamp": "2023-12-05T15:42:03Z",
55+
"ownerReferences": [
56+
{
57+
"apiVersion": "batch/v1",
58+
"kind": "Job",
59+
"name": "sample-job-jg9dw",
60+
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
61+
}
62+
]
63+
},
64+
"priority": 0,
65+
"localQueueName": "user-queue",
66+
"positionInClusterQueue": 1,
67+
"positionInLocalQueue": 1
68+
},
69+
{
70+
"metadata": {
71+
"name": "job-sample-job-t9b8m-4e770",
72+
"namespace": "default",
73+
"creationTimestamp": "2023-12-05T15:42:03Z",
74+
"ownerReferences": [
75+
{
76+
"apiVersion": "batch/v1",
77+
"kind": "Job",
78+
"name": "sample-job-t9b8m",
79+
"uid": "64c26c73-6334-4d13-a1a8-38d99196baa5"
80+
}
81+
]
82+
},
83+
"priority": 0,
84+
"localQueueName": "user-queue",
85+
"positionInClusterQueue": 2,
86+
"positionInLocalQueue": 2
87+
}
88+
]
89+
}
90+
----
91+
+
92+
You can pass the following optional query parameters:
93+
+
94+
`limit <integer>`:: 1000 is the default. Specifies the maximum number of pending workloads that should be fetched.
95+
+
96+
`offset <integer>`:: 0 is the default. Specifies the position of the first pending workload that should be fetched, starting from 0.
97+
98+
. To view only one pending workload starting from position 0 in `ClusterQueue` run the following command:
99+
+
100+
[source,terminal]
101+
----
102+
$ oc get --raw "/apis/visibility.kueue.x-k8s.io/v1beta1/clusterqueues/cluster-queue/pendingworkloads?limit=1&offset=0"
103+
----

0 commit comments

Comments
 (0)