diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kueue-hami-vgpu.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kueue-hami-vgpu.md new file mode 100644 index 00000000..25b311a9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/kueue-hami-vgpu.md @@ -0,0 +1,393 @@ +--- +title: "实验 9: 使用 Kueue 管理 HAMi vGPU 队列" +description: "在 Pod 进入调度器之前，按 vGPU 数量、显存和算力执行配额准入。" +sidebar_label: "实验 9: Kueue + HAMi vGPU" +lab: + level: Advanced + duration: 约 60 分钟 + environment: 单张 NVIDIA Tesla T4 的 Kubernetes 1.36 集群 + authors: + - lixd + verified: "2026-07-13" +tags: + - kueue + - DRA + - vgpu + - quota +toc_max_heading_level: 2 +--- + +本实验把 HAMi GPU 切分与 Kueue 准入控制接到一起。HAMi 为 Pod 分配显存和算力切片，Kueue 在 Job 准入前统计这些切片。你将提交三个相同规格的 Job：前两个正常运行，第三个因为 vGPU、显存和算力配额已经用完而保持挂起。 + +文中的输出来自一套实际环境：Kubernetes 1.36.1、containerd 2.2.4、Kueue 0.18.1、HAMi 2.9.0，GPU 为一张 15 GiB Tesla T4。 + +:::warning 版本相关的 API + +本实验使用 Kueue 0.18.1 提供的 `v1beta2` API 和 `ResourceTransformation` 配置。若使用其他 Kueue 版本，请先核对对应版本的发布说明。 + +::: + +## 你将学到什么 + +- 以 DRA 兼容模式安装 HAMi 2.9.0； +- 验证 `4 GiB / 50%` 扩展资源请求如何转换为 DRA `ResourceClaim`； +- 把 HAMi 按单个 vGPU 表达的显存和算力请求转换成 Kueue 总量配额； +- 在准入阶段挂起超额 Job，避免 Pod 进入调度阶段后才 Pending。 + +## 实验概览 + +```mermaid +%% title: Kueue 与 HAMi vGPU 实验流程 +flowchart LR + Step1["步骤 1
安装 HAMi DRA"] --> Step2["步骤 2
检查设备容量"] + Step2 --> Step3["步骤 3
验证一个 vGPU"] + Step3 --> Step4["步骤 4
配置资源转换"] + Step4 --> Step5["步骤 5
创建队列配额"] + Step5 --> Step6["步骤 6
准入两个 Job"] + Step6 --> Step7["步骤 7
阻塞第三个 Job"] +``` + +## 前提条件 + +- Kubernetes 1.36 集群，已安装 Kueue 0.18.1，并启用 `batch/job` 集成。 +- 一个 NVIDIA GPU 节点。本文验证环境使用 15 GiB Tesla T4。 +- GPU Operator 或等价组件已提供 NVIDIA 驱动和 GPU Feature Discovery 标签。 +- NVIDIA Device Plugin 已关闭。本实验由 HAMi 接管 GPU 设备路径。 +- 有权限安装集群级资源、编辑 Kueue manager 配置的 `kubectl` 和 Helm 环境。 +- 已安装 cert-manager，HAMi DRA webhook 需要用它签发 TLS 证书。 +- [`tutorials/labs/examples/09-kueue-hami-vgpu/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/09-kueue-hami-vgpu) 中的实验清单。 + +如果 GPU Operator 管理 GPU 节点，安装或升级时需关闭它的 device plugin： + +```text +--set devicePlugin.enabled=false +``` + +## 步骤 1: 以 DRA 兼容模式安装 HAMi + +创建实验命名空间。通过 GPU Feature Discovery 标签选择 T4 节点，不要照抄验证环境里的节点名，然后为 HAMi 添加标签： + +```bash +kubectl create namespace hami-kueue-demo +export GPU_NODE=$(kubectl get nodes -l nvidia.com/gpu.product=Tesla-T4 \ + -o jsonpath='{.items[0].metadata.name}') +test -n "${GPU_NODE}" && echo "GPU_NODE=${GPU_NODE}" +kubectl label node "${GPU_NODE}" gpu=on --overwrite +``` + +安装 HAMi 2.9.0，开启 DRA，关闭传统 device plugin： + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +helm install hami hami-charts/hami \ + --namespace hami-system \ + --create-namespace \ + --version 2.9.0 \ + --set dra.enabled=true \ + --set devicePlugin.enabled=false +``` + +:::important + +不要同时开启 HAMi DRA 和传统 device plugin 模式。如果 NVIDIA 驱动直接安装在主机上，而不是由 GPU Operator 管理，还需设置 `hami-dra.drivers.nvidia.containerDriver=false`。 + +::: + +等待三个 DRA 组件启动： + +```bash +kubectl get pods -n hami-system +``` + +```plaintext +NAME READY STATUS +hami-dra-driver-kubelet-plugin-fb4zm 1/1 Running +hami-hami-dra-monitor-7b8df84bd-jsjrd 1/1 Running +hami-hami-dra-webhook-7bb65cbcc5-g5742 1/1 Running +``` + +## 步骤 2: 检查 HAMi 发布的 GPU 容量 + +HAMi 会发布一个 `DeviceClass` 和一个节点级 `ResourceSlice`： + +```bash +kubectl get deviceclass,resourceslice +``` + +```plaintext +NAME AGE +deviceclass.resource.k8s.io/hami-core-gpu.project-hami.io 32m + +NAME NODE DRIVER +resourceslice.resource.k8s.io/lixd-test-gpu-hami-core-gpu.project-hami.io-2drs2 lixd-test-gpu hami-core-gpu.project-hami.io +``` + +查看设备容量： + +```bash +kubectl get resourceslice \ + -o jsonpath='{.items[0].spec.devices[0]}' | python3 -m json.tool +``` + +验证环境中的 T4 包含以下字段： + +```json +{ + "allowMultipleAllocations": true, + "capacity": { + "cores": { "value": "100" }, + "memory": { "value": "15Gi" } + }, + "name": "hami-gpu-0" +} +``` + +`allowMultipleAllocations: true` 表示同一张物理 GPU 可以分配给多个 Claim，直到显存或算力容量用完。 + +## 步骤 3: 验证一个 HAMi vGPU 切片 + +兼容模式允许现有业务继续使用 HAMi 扩展资源。下面的请求表示一个 vGPU、4096 MiB 显存和 50% 算力： + +```bash +kubectl apply -f tutorials/labs/examples/09-kueue-hami-vgpu/01-smoke-pod.yaml +kubectl wait -n hami-kueue-demo \ + --for=condition=Ready pod/hami-compatible-smoke --timeout=5m +``` + +HAMi webhook 会把请求转换成 DRA Claim： + +```bash +kubectl get resourceclaim -n hami-kueue-demo \ + hami-kueue-demo-hami-compatible-smoke-cuda \ + -o jsonpath='{.status.allocation.devices.results[0]}' | python3 -m json.tool +``` + +```json +{ + "consumedCapacity": { + "cores": "50", + "memory": "4Gi" + }, + "device": "hami-gpu-0", + "driver": "hami-core-gpu.project-hami.io" +} +``` + +容器中可以看到切分后的显存上限： + +```bash +kubectl exec -n hami-kueue-demo hami-compatible-smoke -- nvidia-smi +``` + +```plaintext +| 0 Tesla T4 ... | 0MiB / 4096MiB | 0% Default | +``` + +删除 smoke Pod，避免它影响后面的队列测试： + +```bash +kubectl delete pod -n hami-kueue-demo hami-compatible-smoke +``` + +## 步骤 4: 配置 Kueue 资源转换 + +HAMi 的 `gpumem` 和 `gpucores` 按单个 vGPU 表达，Kueue 需要统计总量。两个相同 Job 各申请一个 vGPU、4096 MiB 和 50% 算力时，总用量为： + +```text +vGPU 实例数：2 +显存总量： 2 x 4096 MiB = 8192 MiB +算力总量： 2 x 50 = 100 +``` + +编辑 Kueue manager 配置： + +```bash +kubectl edit configmap kueue-manager-config -n kueue-system +``` + +在 `controller_manager_config.yaml` 的现有 `Configuration` 文档中加入 `resources.transformations`： + +```yaml +apiVersion: config.kueue.x-k8s.io/v1beta2 +kind: Configuration +integrations: + frameworks: + - batch/job +resources: + transformations: + - input: nvidia.com/gpumem + strategy: Replace + outputs: + nvidia.com/total-gpumem: 1 + multiplyBy: nvidia.com/gpu + - input: nvidia.com/gpucores + strategy: Replace + outputs: + nvidia.com/total-gpucores: 1 + multiplyBy: nvidia.com/gpu +``` + +保留原有配置的其余部分。重启 Kueue，并等待 rollout 完成： + +```bash +kubectl rollout restart deployment/kueue-controller-manager -n kueue-system +kubectl rollout status deployment/kueue-controller-manager -n kueue-system +``` + +```plaintext +deployment "kueue-controller-manager" successfully rolled out +``` + +`Replace` 会从 Kueue 记账中移除按设备表达的输入资源；`multiplyBy: nvidia.com/gpu` 根据 vGPU 实例数计算显存和算力总量。 + +## 步骤 5: 创建 Kueue 配额 + +本实验的队列允许两个 vGPU 实例、8192 MiB 显存总量和 100 点算力总量： + +```bash +kubectl apply -f tutorials/labs/examples/09-kueue-hami-vgpu/02-queues.yaml +kubectl get resourceflavor,clusterqueue +kubectl get localqueue -n hami-kueue-demo +``` + +```plaintext +NAME AGE +resourceflavor.kueue.x-k8s.io/hami-t4 8s + +NAME COHORT PENDING WORKLOADS ADMITTED WORKLOADS +clusterqueue.kueue.x-k8s.io/hami-cq 0 0 + +NAME CLUSTERQUEUE PENDING WORKLOADS ADMITTED WORKLOADS +localqueue.kueue.x-k8s.io/hami-queue hami-cq 0 0 +``` + +显存配额以 MiB 为单位，与工作负载里的 `nvidia.com/gpumem: 4096` 一致。`ResourceFlavor` 的节点标签必须与 GPU 节点匹配；如果不是 T4，请修改清单中的 `Tesla-T4`。 + +## 步骤 6: 准入两个 vGPU Job + +创建三个相同规格的 Job： + +```bash +kubectl apply -f tutorials/labs/examples/09-kueue-hami-vgpu/03-jobs.yaml +kubectl get job,workload -n hami-kueue-demo +``` + +```plaintext +NAME STATUS +job.batch/hami-kueue-running-a Running +job.batch/hami-kueue-running-b Running +job.batch/hami-kueue-pending-c Suspended + +NAME QUEUE RESERVED IN ADMITTED +workload.kueue.x-k8s.io/job-hami-kueue-running-a-59997 hami-queue hami-cq True +workload.kueue.x-k8s.io/job-hami-kueue-running-b-9d737 hami-queue hami-cq True +workload.kueue.x-k8s.io/job-hami-kueue-pending-c-d854d hami-queue +``` + +查看任意一个已准入 Workload： + +```bash +kubectl get workload -n hami-kueue-demo \ + -l kueue.x-k8s.io/job-name=hami-kueue-running-a \ + -o jsonpath='{.items[0].status.admission.podSetAssignments[0].resourceUsage}' \ + | python3 -m json.tool +``` + +```json +{ + "nvidia.com/gpu": "1", + "nvidia.com/total-gpucores": "50", + "nvidia.com/total-gpumem": "4096" +} +``` + +HAMi 分配 DRA Claim、Pod 进入常规调度之前，Kueue 已经扣除了三项配额。 + +## 步骤 7: 验证第三个 Job 留在队列中 + +查看 ClusterQueue 用量： + +```bash +kubectl get clusterqueue hami-cq -o yaml +``` + +```yaml +status: + admittedWorkloads: 2 + flavorsUsage: + - name: hami-t4 + resources: + - name: nvidia.com/gpu + total: "2" + - name: nvidia.com/total-gpucores + total: "100" + - name: nvidia.com/total-gpumem + total: "8192" + pendingWorkloads: 1 +``` + +Pending Workload 同时记录了转换后的请求和未准入原因： + +```bash +kubectl get workload -n hami-kueue-demo \ + -l kueue.x-k8s.io/job-name=hami-kueue-pending-c -o yaml +``` + +```yaml +status: + conditions: + - reason: Pending + message: >- + couldn't assign flavors to pod set main: insufficient unused quota for nvidia.com/gpu in flavor hami-t4, 1 more needed, insufficient unused quota for nvidia.com/total-gpucores in flavor hami-t4, 50 more needed + + + resourceRequests: + - resources: + nvidia.com/gpu: "1" + nvidia.com/total-gpucores: "50" + nvidia.com/total-gpumem: "4096" +``` + +第三个 Job 保持 `Suspended`，不会创建 Pod 去竞争已经用完的 GPU 容量。 + +## 清理 + +删除工作负载和队列资源： + +```bash +kubectl delete -f tutorials/labs/examples/09-kueue-hami-vgpu/03-jobs.yaml +kubectl delete -f tutorials/labs/examples/09-kueue-hami-vgpu/02-queues.yaml +kubectl delete namespace hami-kueue-demo +``` + +从 `kueue-manager-config` 删除两项 `resources.transformations`，然后重启 Kueue： + +```bash +kubectl edit configmap kueue-manager-config -n kueue-system +kubectl rollout restart deployment/kueue-controller-manager -n kueue-system +kubectl rollout status deployment/kueue-controller-manager -n kueue-system +``` + +如果集群只用于本实验，可以卸载 HAMi： + +```bash +helm uninstall hami -n hami-system +kubectl delete namespace hami-system +``` + +## 本实验验证了什么 + +| 结论 | 证据 | +| --- | --- | +| HAMi 把扩展资源请求转换成 DRA 分配 | 生成的 `ResourceClaim` 从 `hami-gpu-0` 消耗 `4Gi` 显存和 `50` cores | +| vGPU 显存上限进入了容器 | `nvidia-smi` 显示 4096 MiB 上限 | +| Kueue 把单个 vGPU 资源折算成总量 | 每个已准入 Workload 使用一个 vGPU、4096 MiB 总显存和 50 点总算力 | +| 队列准入阻止了超额任务 | 两个 Job 运行，第三个因配额不足保持挂起 | + +## 后续练习 + +- 只调整显存配额，观察最先触发阻塞的配额维度。 +- 为不同团队创建独立 `ClusterQueue`，分配不同的 GPU 预算。 +- 对比 [实验 4](./hami-dra.md) 中直接使用 `ResourceClaim` 的原生 DRA 流程。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index ba78b71b..2aa8e883 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -19,4 +19,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; -每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。 +每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。实验 9 使用 Kueue 准入控制限制 HAMi vGPU 数量、显存和算力配额。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml index 5ed0821e..f01bd7e1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml @@ -19,6 +19,9 @@ "k3s": label: "k3s" permalink: "/k-3-s" +"kueue": + label: "Kueue" + permalink: "/kueue" "nvidia": label: "nvidia" permalink: "/nvidia" @@ -28,6 +31,9 @@ "queue": label: "queue" permalink: "/queue" +"quota": + label: "配额" + permalink: "/quota" "vgpu": label: "vgpu" permalink: "/vgpu" diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index 8b1fdfd0..efe8ddca 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -51,6 +51,11 @@ module.exports = { id: "labs/volcano-vgpu-gang-queue", customProps: { level: "Advanced", duration: "about 60 minutes" }, }, + { + type: "doc", + id: "labs/kueue-hami-vgpu", + customProps: { level: "Advanced", duration: "about 60 minutes" }, + }, ], }, ], diff --git a/tutorials/labs/examples/09-kueue-hami-vgpu/01-smoke-pod.yaml b/tutorials/labs/examples/09-kueue-hami-vgpu/01-smoke-pod.yaml new file mode 100644 index 00000000..a83d4b90 --- /dev/null +++ b/tutorials/labs/examples/09-kueue-hami-vgpu/01-smoke-pod.yaml @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Pod +metadata: + name: hami-compatible-smoke + namespace: hami-kueue-demo +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvcr.io/nvidia/cuda:12.6.0-base-ubuntu24.04 + imagePullPolicy: IfNotPresent + command: ["bash", "-c", "nvidia-smi && sleep 3600"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 50 diff --git a/tutorials/labs/examples/09-kueue-hami-vgpu/02-queues.yaml b/tutorials/labs/examples/09-kueue-hami-vgpu/02-queues.yaml new file mode 100644 index 00000000..08cb3638 --- /dev/null +++ b/tutorials/labs/examples/09-kueue-hami-vgpu/02-queues.yaml @@ -0,0 +1,36 @@ +apiVersion: kueue.x-k8s.io/v1beta2 +kind: ResourceFlavor +metadata: + name: hami-t4 +spec: + nodeLabels: + nvidia.com/gpu.product: Tesla-T4 +--- +apiVersion: kueue.x-k8s.io/v1beta2 +kind: ClusterQueue +metadata: + name: hami-cq +spec: + namespaceSelector: {} + resourceGroups: + - coveredResources: + - nvidia.com/gpu + - nvidia.com/total-gpumem + - nvidia.com/total-gpucores + flavors: + - name: hami-t4 + resources: + - name: nvidia.com/gpu + nominalQuota: 2 + - name: nvidia.com/total-gpumem + nominalQuota: 8192 + - name: nvidia.com/total-gpucores + nominalQuota: 100 +--- +apiVersion: kueue.x-k8s.io/v1beta2 +kind: LocalQueue +metadata: + name: hami-queue + namespace: hami-kueue-demo +spec: + clusterQueue: hami-cq diff --git a/tutorials/labs/examples/09-kueue-hami-vgpu/03-jobs.yaml b/tutorials/labs/examples/09-kueue-hami-vgpu/03-jobs.yaml new file mode 100644 index 00000000..02c8870b --- /dev/null +++ b/tutorials/labs/examples/09-kueue-hami-vgpu/03-jobs.yaml @@ -0,0 +1,74 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: hami-kueue-running-a + namespace: hami-kueue-demo + labels: + kueue.x-k8s.io/queue-name: hami-queue +spec: + suspend: true + parallelism: 1 + completions: 1 + template: + spec: + restartPolicy: Never + containers: + - name: cuda + image: nvcr.io/nvidia/cuda:12.6.0-base-ubuntu24.04 + imagePullPolicy: IfNotPresent + command: ["bash", "-c", "nvidia-smi && echo HAMI_KUEUE_RUNNING_A && sleep 3600"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 50 +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: hami-kueue-running-b + namespace: hami-kueue-demo + labels: + kueue.x-k8s.io/queue-name: hami-queue +spec: + suspend: true + parallelism: 1 + completions: 1 + template: + spec: + restartPolicy: Never + containers: + - name: cuda + image: nvcr.io/nvidia/cuda:12.6.0-base-ubuntu24.04 + imagePullPolicy: IfNotPresent + command: ["bash", "-c", "nvidia-smi && echo HAMI_KUEUE_RUNNING_B && sleep 3600"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 50 +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: hami-kueue-pending-c + namespace: hami-kueue-demo + labels: + kueue.x-k8s.io/queue-name: hami-queue +spec: + suspend: true + parallelism: 1 + completions: 1 + template: + spec: + restartPolicy: Never + containers: + - name: cuda + image: nvcr.io/nvidia/cuda:12.6.0-base-ubuntu24.04 + imagePullPolicy: IfNotPresent + command: ["bash", "-c", "nvidia-smi && echo HAMI_KUEUE_PENDING_C && sleep 3600"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 4096 + nvidia.com/gpucores: 50 diff --git a/tutorials/labs/kueue-hami-vgpu.md b/tutorials/labs/kueue-hami-vgpu.md new file mode 100644 index 00000000..5124e27f --- /dev/null +++ b/tutorials/labs/kueue-hami-vgpu.md @@ -0,0 +1,393 @@ +--- +title: "Lab 9: Queue HAMi vGPU Workloads with Kueue" +description: "Enforce vGPU count, memory, and compute quotas for HAMi workloads before Pods reach the scheduler." +sidebar_label: "Lab 9: Kueue + HAMi vGPU" +lab: + level: Advanced + duration: about 60 minutes + environment: Kubernetes 1.36 cluster with one NVIDIA Tesla T4 + authors: + - lixd + verified: "2026-07-13" +tags: + - kueue + - dra + - vgpu + - quota +toc_max_heading_level: 2 +--- + +This lab combines HAMi GPU slicing with Kueue admission control. HAMi gives each Pod a slice of GPU memory and compute capacity; Kueue accounts for those slices before admitting the Job. You will admit two identical Jobs and verify that a third remains suspended when the queue reaches its vGPU, memory, and compute quotas. + +The captured output comes from Kubernetes 1.36.1, containerd 2.2.4, Kueue 0.18.1, and HAMi 2.9.0 on a node with one 15 GiB Tesla T4. + +:::warning Version-specific APIs + +This lab uses Kueue `v1beta2` APIs and the `ResourceTransformation` configuration available in Kueue 0.18.1. Check the Kueue release notes before applying the configuration to a different version. + +::: + +## What You'll Learn + +- install HAMi 2.9.0 in DRA compatibility mode; +- verify that a `4 GiB / 50%` extended-resource request becomes a DRA `ResourceClaim`; +- transform per-vGPU HAMi memory and compute requests into total Kueue quota usage; and +- keep excess Jobs suspended at admission time instead of letting their Pods become scheduler-level Pending. + +## Lab Overview + +```mermaid +%% title: Kueue and HAMi vGPU Lab Flow +flowchart LR + Step1["Step 1
Install HAMi DRA"] --> Step2["Step 2
Verify device capacity"] + Step2 --> Step3["Step 3
Test one vGPU"] + Step3 --> Step4["Step 4
Configure transformations"] + Step4 --> Step5["Step 5
Create queue quotas"] + Step5 --> Step6["Step 6
Admit two Jobs"] + Step6 --> Step7["Step 7
Block the third Job"] +``` + +## Prerequisites + +- A Kubernetes 1.36 cluster with Kueue 0.18.1 installed and the `batch/job` integration enabled. +- One NVIDIA GPU node. The verified environment uses a 15 GiB Tesla T4. +- GPU Operator or an equivalent installation that provides the NVIDIA driver and GPU Feature Discovery labels. +- NVIDIA Device Plugin disabled because HAMi owns the GPU device path in this lab. +- `kubectl` and Helm access with permission to install cluster-scoped resources and edit the Kueue manager configuration. +- cert-manager installed; the HAMi DRA webhook uses it to provision TLS certificates. +- The manifests from [`tutorials/labs/examples/09-kueue-hami-vgpu/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/09-kueue-hami-vgpu). + +If GPU Operator manages the node, install or upgrade it with its device plugin disabled: + +```text +--set devicePlugin.enabled=false +``` + +## Step 1: Install HAMi in DRA Compatibility Mode + +Create the namespace used throughout the lab. Select the T4 node from its GPU Feature Discovery label instead of copying the verified environment's node name, then label it for HAMi: + +```bash +kubectl create namespace hami-kueue-demo +export GPU_NODE=$(kubectl get nodes -l nvidia.com/gpu.product=Tesla-T4 \ + -o jsonpath='{.items[0].metadata.name}') +test -n "${GPU_NODE}" && echo "GPU_NODE=${GPU_NODE}" +kubectl label node "${GPU_NODE}" gpu=on --overwrite +``` + +Install HAMi 2.9.0 with DRA enabled and the traditional device plugin disabled: + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi/ +helm repo update + +helm install hami hami-charts/hami \ + --namespace hami-system \ + --create-namespace \ + --version 2.9.0 \ + --set dra.enabled=true \ + --set devicePlugin.enabled=false +``` + +:::important + +Do not enable HAMi's DRA and traditional device-plugin modes at the same time. If the NVIDIA driver is installed directly on the host rather than by GPU Operator, also set `hami-dra.drivers.nvidia.containerDriver=false`. + +::: + +Wait for the three DRA components: + +```bash +kubectl get pods -n hami-system +``` + +```plaintext +NAME READY STATUS +hami-dra-driver-kubelet-plugin-fb4zm 1/1 Running +hami-hami-dra-monitor-7b8df84bd-jsjrd 1/1 Running +hami-hami-dra-webhook-7bb65cbcc5-g5742 1/1 Running +``` + +## Step 2: Inspect the Published GPU Capacity + +HAMi publishes a `DeviceClass` and a node-local `ResourceSlice`: + +```bash +kubectl get deviceclass,resourceslice +``` + +```plaintext +NAME AGE +deviceclass.resource.k8s.io/hami-core-gpu.project-hami.io 32m + +NAME NODE DRIVER +resourceslice.resource.k8s.io/lixd-test-gpu-hami-core-gpu.project-hami.io-2drs2 lixd-test-gpu hami-core-gpu.project-hami.io +``` + +Inspect the device capacity: + +```bash +kubectl get resourceslice \ + -o jsonpath='{.items[0].spec.devices[0]}' | python3 -m json.tool +``` + +The verified T4 reported these relevant fields: + +```json +{ + "allowMultipleAllocations": true, + "capacity": { + "cores": { "value": "100" }, + "memory": { "value": "15Gi" } + }, + "name": "hami-gpu-0" +} +``` + +`allowMultipleAllocations: true` lets multiple claims consume capacity from the same physical GPU until memory or compute capacity is exhausted. + +## Step 3: Verify One HAMi vGPU Slice + +Compatibility mode lets an existing workload keep using HAMi extended resources. The request below means one vGPU with 4096 MiB of memory and 50% compute capacity: + +```bash +kubectl apply -f tutorials/labs/examples/09-kueue-hami-vgpu/01-smoke-pod.yaml +kubectl wait -n hami-kueue-demo \ + --for=condition=Ready pod/hami-compatible-smoke --timeout=5m +``` + +HAMi's webhook converts that request into a DRA claim. Inspect the generated claim: + +```bash +kubectl get resourceclaim -n hami-kueue-demo \ + hami-kueue-demo-hami-compatible-smoke-cuda \ + -o jsonpath='{.status.allocation.devices.results[0]}' | python3 -m json.tool +``` + +```json +{ + "consumedCapacity": { + "cores": "50", + "memory": "4Gi" + }, + "device": "hami-gpu-0", + "driver": "hami-core-gpu.project-hami.io" +} +``` + +The container sees the sliced memory ceiling: + +```bash +kubectl exec -n hami-kueue-demo hami-compatible-smoke -- nvidia-smi +``` + +```plaintext +| 0 Tesla T4 ... | 0MiB / 4096MiB | 0% Default | +``` + +Remove the smoke Pod so it does not consume GPU capacity during the queue test: + +```bash +kubectl delete pod -n hami-kueue-demo hami-compatible-smoke +``` + +## Step 4: Configure Kueue Resource Transformations + +HAMi expresses `gpumem` and `gpucores` per vGPU. Kueue needs total usage. For example, two Jobs that each request one vGPU, 4096 MiB, and 50% compute consume: + +```text +vGPU instances: 2 +total memory: 2 x 4096 MiB = 8192 MiB +total compute: 2 x 50 = 100 +``` + +Edit the Kueue manager configuration: + +```bash +kubectl edit configmap kueue-manager-config -n kueue-system +``` + +Add `resources.transformations` to the existing `Configuration` document in `controller_manager_config.yaml`: + +```yaml +apiVersion: config.kueue.x-k8s.io/v1beta2 +kind: Configuration +integrations: + frameworks: + - batch/job +resources: + transformations: + - input: nvidia.com/gpumem + strategy: Replace + outputs: + nvidia.com/total-gpumem: 1 + multiplyBy: nvidia.com/gpu + - input: nvidia.com/gpucores + strategy: Replace + outputs: + nvidia.com/total-gpucores: 1 + multiplyBy: nvidia.com/gpu +``` + +Preserve the rest of the existing configuration. Restart Kueue and wait for it to become available: + +```bash +kubectl rollout restart deployment/kueue-controller-manager -n kueue-system +kubectl rollout status deployment/kueue-controller-manager -n kueue-system +``` + +```plaintext +deployment "kueue-controller-manager" successfully rolled out +``` + +The `Replace` strategy removes the per-device input from Kueue accounting. `multiplyBy: nvidia.com/gpu` creates total memory and compute requests based on the number of requested vGPU instances. + +## Step 5: Create the Kueue Quotas + +The queue allows two vGPU instances, 8192 MiB of total GPU memory, and 100 total compute points: + +```bash +kubectl apply -f tutorials/labs/examples/09-kueue-hami-vgpu/02-queues.yaml +kubectl get resourceflavor,clusterqueue +kubectl get localqueue -n hami-kueue-demo +``` + +```plaintext +NAME AGE +resourceflavor.kueue.x-k8s.io/hami-t4 8s + +NAME COHORT PENDING WORKLOADS ADMITTED WORKLOADS +clusterqueue.kueue.x-k8s.io/hami-cq 0 0 + +NAME CLUSTERQUEUE PENDING WORKLOADS ADMITTED WORKLOADS +localqueue.kueue.x-k8s.io/hami-queue hami-cq 0 0 +``` + +The memory quota uses MiB, matching `nvidia.com/gpumem: 4096` in the workload. The `ResourceFlavor` node label must match the label on your GPU node; adjust `Tesla-T4` if you use another model. + +## Step 6: Admit Two vGPU Jobs + +Create three identical Jobs. Applying one file keeps their specifications exactly the same apart from the names: + +```bash +kubectl apply -f tutorials/labs/examples/09-kueue-hami-vgpu/03-jobs.yaml +kubectl get job,workload -n hami-kueue-demo +``` + +```plaintext +NAME STATUS +job.batch/hami-kueue-running-a Running +job.batch/hami-kueue-running-b Running +job.batch/hami-kueue-pending-c Suspended + +NAME QUEUE RESERVED IN ADMITTED +workload.kueue.x-k8s.io/job-hami-kueue-running-a-59997 hami-queue hami-cq True +workload.kueue.x-k8s.io/job-hami-kueue-running-b-9d737 hami-queue hami-cq True +workload.kueue.x-k8s.io/job-hami-kueue-pending-c-d854d hami-queue +``` + +Inspect either admitted Workload: + +```bash +kubectl get workload -n hami-kueue-demo \ + -l kueue.x-k8s.io/job-name=hami-kueue-running-a \ + -o jsonpath='{.items[0].status.admission.podSetAssignments[0].resourceUsage}' \ + | python3 -m json.tool +``` + +```json +{ + "nvidia.com/gpu": "1", + "nvidia.com/total-gpucores": "50", + "nvidia.com/total-gpumem": "4096" +} +``` + +Kueue has charged all three quota dimensions before HAMi allocates the DRA claim and before the Pod reaches normal scheduling. + +## Step 7: Verify the Third Job Stays Queued + +Inspect the ClusterQueue usage: + +```bash +kubectl get clusterqueue hami-cq -o yaml +``` + +```yaml +status: + admittedWorkloads: 2 + flavorsUsage: + - name: hami-t4 + resources: + - name: nvidia.com/gpu + total: "2" + - name: nvidia.com/total-gpucores + total: "100" + - name: nvidia.com/total-gpumem + total: "8192" + pendingWorkloads: 1 +``` + +The pending Workload records both its transformed request and why it was not admitted: + +```bash +kubectl get workload -n hami-kueue-demo \ + -l kueue.x-k8s.io/job-name=hami-kueue-pending-c -o yaml +``` + +```yaml +status: + conditions: + - reason: Pending + message: >- + couldn't assign flavors to pod set main: insufficient unused quota for nvidia.com/gpu in flavor hami-t4, 1 more needed, insufficient unused quota for nvidia.com/total-gpucores in flavor hami-t4, 50 more needed + + + resourceRequests: + - resources: + nvidia.com/gpu: "1" + nvidia.com/total-gpucores: "50" + nvidia.com/total-gpumem: "4096" +``` + +The third Job remains `Suspended`; it does not create a Pod that competes for already exhausted GPU capacity. + +## Cleanup + +Delete the workloads and queue resources: + +```bash +kubectl delete -f tutorials/labs/examples/09-kueue-hami-vgpu/03-jobs.yaml +kubectl delete -f tutorials/labs/examples/09-kueue-hami-vgpu/02-queues.yaml +kubectl delete namespace hami-kueue-demo +``` + +Remove the two `resources.transformations` entries from `kueue-manager-config`, then restart Kueue: + +```bash +kubectl edit configmap kueue-manager-config -n kueue-system +kubectl rollout restart deployment/kueue-controller-manager -n kueue-system +kubectl rollout status deployment/kueue-controller-manager -n kueue-system +``` + +If this cluster is only for the lab, uninstall HAMi: + +```bash +helm uninstall hami -n hami-system +kubectl delete namespace hami-system +``` + +## What This Lab Proved + +| Claim | Evidence | +| --- | --- | +| HAMi converted an extended-resource request into a DRA allocation | The generated `ResourceClaim` consumed `4Gi` memory and `50` cores from `hami-gpu-0` | +| The vGPU memory limit reached the container | `nvidia-smi` reported a 4096 MiB ceiling | +| Kueue accounted for per-vGPU resources as totals | Each admitted Workload used one vGPU, 4096 total MiB, and 50 total compute points | +| Queue admission prevented oversubscription | Two Jobs ran while the third remained suspended with an insufficient-quota condition | + +## Next Steps + +- Change only the memory quota to observe which quota dimension blocks admission first. +- Add separate `ClusterQueue` objects for teams that need different GPU budgets. +- Compare this compatibility path with the native `ResourceClaim` workflow in [Lab 4](./hami-dra.md). diff --git a/tutorials/overview.md b/tutorials/overview.md index 42126285..4da882fa 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -17,4 +17,4 @@ Background knowledge that the labs build on. ## Labs - Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. + Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. Lab 9 uses Kueue admission control to enforce HAMi vGPU count, memory, and compute quotas.