feat(nvidia): Vulkan vGPU support — memory partitioning via vulkan-layer + manifest auto-mount + admission webhook env injection

[100milliongold]

Summary

Adds Vulkan vGPU support to HAMi so that Vulkan workloads (Isaac Sim, ray tracing, GPU-accelerated rendering, etc.) honor the same per-container memory limit that HAMi already enforces for CUDA.

Three coordinated layers, all opt-in via the hami.io/vulkan: "true" pod annotation:

1. HAMi-core Vulkan layer — hooks vkAllocateMemory to enforce CUDA_DEVICE_MEMORY_LIMIT_0. The Vulkan implicit-layer manifest uses enable_environment: HAMI_VULKAN_ENABLE=1 so the layer is loaded only when the pod opts in.
2. Device-plugin — bind-mounts the Vulkan implicit-layer manifest from the host into the container at /etc/vulkan/implicit_layer.d/hami.json.
3. Admission webhook — when the pod has hami.io/vulkan: "true" and requests a GPU resource, injects HAMI_VULKAN_ENABLE=1 and merges graphics into NVIDIA_DRIVER_CAPABILITIES so the runtime exposes the NVIDIA Vulkan ICD.

Pods without the annotation are unaffected — bit-identical to current behavior.

Why

HAMi currently enforces vGPU memory limits only for CUDA workloads. Vulkan applications bypass the limit because vkAllocateMemory is not on the CUDA path. We hit this in production with Isaac Sim — Kit allocates memory through Vulkan, ignored the requested partition, and OOM'd the host. Hooking vkAllocateMemory in the HAMi-core layer closes the gap.

The opt-in annotation is intentional:

• Avoids forcing the graphics capability on every CUDA pod (would change runtime behavior cluster-wide).
• The enable_environment guard means the layer doesn't load even if the manifest happens to be mounted, when the env isn't set.
• Existing CUDA workloads remain unchanged.

What changed

File / Area	Change
libvgpu submodule	→ xiilab/HAMi-core@8d4f712 (Vulkan layer with vkAllocateMemory hook, cuMemFree[Async] untracked-pointer fallback, cuMemGetInfo_v2 OptiX crash fix). A companion PR against Project-HAMi/HAMi-core is needed before this can be merged upstream — happy to open it once direction is agreed.
docker/Dockerfile	Install libvulkan-dev in the nvbuild stage; copy hami.json into the runtime image at /k8s-vgpu/lib/nvidia/vulkan/implicit_layer.d/hami.json
pkg/device-plugin/nvidiadevice/nvinternal/plugin/server.go	When /usr/local/vgpu/vulkan/implicit_layer.d/hami.json exists on the host, append a bind-mount into the container's Allocate response. Idempotent and side-effect-free when the file is absent.
pkg/device/nvidia/device.go	New applyVulkanAnnotation: when the pod carries hami.io/vulkan: "true", sets HAMI_VULKAN_ENABLE=1 and merges graphics into NVIDIA_DRIVER_CAPABILITIES. Called only when the container actually requests a GPU resource.
pkg/device/nvidia/device_test.go	TDD coverage for env injection: no-annotation no-op, capability merge, idempotency, edge cases (existing caps, empty caps, etc.).
examples/nvidia/vulkan_example.yaml	Minimal usage sample.
docs/vulkan-vgpu-support.md	English usage guide.
docs/vulkan-vgpu-support_cn.md	Chinese translation.
docs/vulkan-vgpu-e2e-checklist.md	Manual E2E verification checklist.

How it works

1. The HAMi-core Vulkan layer hooks vkAllocateMemory to enforce the per-container memory limit set by HAMi-core's existing CUDA limit code (same CUDA_DEVICE_MEMORY_LIMIT_0 env).
2. The device-plugin mounts the implicit-layer manifest into the container so the Vulkan loader picks it up automatically.
3. The manifest's enable_environment: HAMI_VULKAN_ENABLE=1 guard means the layer isn't activated unless the env is set.
4. The admission webhook reads hami.io/vulkan: "true", sets the gating env, and merges graphics so the NVIDIA runtime exposes the Vulkan ICD libraries.

Test plan

• go test ./pkg/device/nvidia/... — env injection unit tests pass.
• make docker builds the image with libvulkan-dev and ships hami.json.
• Deploy pod with hami.io/vulkan: "true" annotation → HAMI_VULKAN_ENABLE=1 env present, NVIDIA_DRIVER_CAPABILITIES contains graphics, /etc/vulkan/implicit_layer.d/hami.json mounted.
• Deploy pod without the annotation → unmodified (regression check).
• E2E: ran a Vulkan workload (Isaac Sim) with nvidia.com/gpumem limit; the Kit boot log reports the exact partition size and the workload is held to it.

Verified on

• Hardware: NVIDIA RTX 6000 Ada × 2 (driver 550-series).
• K8s: v1.34.3.
• Integration: tested with both stock HAMi (3-tier: webhook + scheduler + device-plugin) and a webhook-only deployment co-existing with Volcano scheduler. The Vulkan changes are orthogonal to scheduling — they only depend on the webhook + device-plugin path.

Compatibility / Breaking changes

• None for existing CUDA workloads — the Vulkan code paths are gated behind the annotation and the enable_environment runtime guard.
• New container env (HAMI_VULKAN_ENABLE) and new mount path (/etc/vulkan/implicit_layer.d/hami.json) are added only for opted-in pods.

Notes for reviewers

• The submodule change to xiilab/HAMi-core@vulkan-layer is the only blocker for upstream merge; happy to open the companion PR against Project-HAMi/HAMi-core once direction is agreed (e.g. accept as a new branch, fold into a release, or restructure as a build flag).
• This branch carries a few internal planning files under docs/superpowers/ (Korean-language design and implementation plans) that I can drop in a cleanup commit if reviewers prefer a leaner diff.
• The webhook code at pkg/scheduler/webhook.go:64-69 (the existing scheduler-name skip check) has a known operator-precedence issue in v2.8.x that fix: Add option for overwrite schedulerName #1163 fixed on master — this PR does not touch that block.
• The device-plugin change is intentionally null-safe: if the host doesn't have hami.json (e.g. the user opted out of running the manifest installer), the Allocate response is unchanged.

Happy to split into smaller PRs (HAMi-core layer / Dockerfile / device-plugin mount / webhook env / docs) if that's easier to review.

[hami-robot]

Welcome @100milliongold! It looks like this is your first PR to Project-HAMi/HAMi 🎉

[gemini-code-assist]

Code Review

This pull request introduces Vulkan vGPU partitioning support by updating the libvgpu submodule, configuring Docker images with Vulkan dependencies, and implementing admission webhook logic to inject necessary environment variables and manifest mounts. The changes also include comprehensive design specifications and user documentation. Review feedback suggests expanding manifest mounting to support MIG devices, improving the robustness of the graphics capability merging logic, and optimizing environment variable processing in the admission controller for better efficiency.

[100milliongold]

Step B (HAMi-core hook hardening) complete

HAMi-core PR #182 added NULL pointer guards to CUDA hooks:

• cuMemAlloc_v2 (88143ab)
• cuMemAllocManaged (275ba3d)
• cuMemAllocPitch_v2 (01a58f1)
• cuMemHostRegister_v2 (7dcb5a4) — also drops vestigial cuCtxGetDevice
• cuMemHostAlloc / cuCtxGetDevice — tests-only (already correct)

Pattern matches the existing cuMemGetInfo_v2 fix (commit 03f99d7).

The libvgpu submodule pointer is bumped to the new HAMi-core SHA (commit 8cfcebb).

isaac-launchable baseline preserved (5/5 runheadless.sh alive). 9/9 unit tests pass + 4/4 manual NULL stress tests in pod under LD_PRELOAD pass without crash.

Step C (Vulkan layer compat for Isaac Sim Kit init under LD_PRELOAD) follows in a separate plan.

Spec: docs/superpowers/specs/2026-04-28-hami-isolation-isaac-sim-design.md
Plan: docs/superpowers/plans/2026-04-28-hami-isolation-step-b-cuda-hook-hardening.md

[100milliongold]

Step C — Vulkan layer split (libvgpu_vk.so)

HAMi-core PR #182 redesigned Step C: libvgpu.so is now HAMi-core only, and a new libvgpu_vk.so holds the Vulkan implicit layer. Activation moves entirely to the manifest path, removing the LD_PRELOAD/Vulkan-loader collision surface that bit us on 2026-04-28 (see docs/superpowers/notes/2026-04-28-vk-trace-isaac-sim.md in the submodule).

The libvgpu submodule pointer is bumped from 7dcb5a4 (Step B end) to 65930f4 (Step C redesign end).

Verification on ws-node074, isaac-launchable-0

• LD_PRELOAD libvgpu.so without manifest: 5/5 runheadless.sh exit=124 alive, crash=0, listen :49100 — the regression class introduced by the 2026-04-28 first attempt is structurally gone in this architecture.
• LD_PRELOAD libvgpu.so + hami.json manifest installed in pod: 5/5 alive, manifest enumerated by Vulkan loader, NVML hook clamp 44 GiB → 23 GiB confirmed via nvidia-smi.
• HAMI_VK_TRACE instrumentation showed 0 lines on the manifest path during this run; the embedded Conan Vulkan loader inside Kit's runheadless did not invoke our GIPA wrappers in this test. Partition enforcement on the Vulkan path will be confirmed in Step D's 4-path verification once the webhook installs the manifest at the host level alongside libvgpu_vk.so.
• Production /usr/local/vgpu/libvgpu.so restored to backup md5 8f889313 after verification — pod baseline exit=124 crash=0 listen=1 confirmed post-restore.

What's next

• Step D (separate plan) will productionize the activation: webhook-injected LD_PRELOAD env + DaemonSet-installed hami.json + host-installed libvgpu_vk.so, plus the 4-path partition-enforcement verification (NVML, CUDA, Vulkan memory query, Vulkan allocate).
• The original Step C "Tasks 1+2" (cache first next-gipa, GIPA/GDPA fallback for unknown handles) stay reverted and become a follow-up PR after Step D end-to-end verification; the trace evidence already on the submodule branch is enough background for that follow-up.

Spec / Plan

• Spec: docs/superpowers/specs/2026-04-29-step-c-redesign-vk-so-split.md
• Plan: docs/superpowers/plans/2026-04-29-step-c-vk-so-split.md

[100milliongold]

Step D — Vulkan opt-in production activation + 4-path 검증

Step C 의 libvgpu_vk.so 분리 산출물을 production opt-in path 에서 활성화. ws-node074 isaac-launchable-0 에서 NVML / CUDA / Vulkan-memory-query / Vulkan-allocate 4 path 모두 partition enforce 검증 완료.

Commits (HAMi parent, this push)

SHA	Message
9f06dc0	chore(runtime): Step D — update hami-vulkan-manifest CM to libvgpu_vk.so
c5d4b89	feat(nvidia): inject Vulkan manifest hostPath mount on opt-in pods
8542d63	build(docker): set CI_COMMIT_SHA in nvbuild stage to skip git describe
c15889a	feat(nvidia): also inject libvgpu_vk.so hostPath into Vulkan opt-in pods
a55a5a5	chore(libvgpu): bump HAMi-core for Step D — re-apply Step C Tasks 1+2 in split arch

Companion changes

• libvgpu fork (xiilab/vulkan-layer): cherry-picked Step C Tasks 1+2 (996cb22, eea2beb) on top of the previously-pushed Step C redesign — they were originally reverted due to LD_PRELOAD-only regression on the combined-libvgpu.so build, but the Step C redesign moved Vulkan code into a separate libvgpu_vk.so that is manifest-activated only, so that regression scenario can no longer occur. Path 4 (vkAllocateMemory over-budget OOM) requires these hooks because NVIDIA's vkEnumerateDeviceExtensionProperties returns LayerNotPresent for our layer name and breaks vkCreateDevice chain assembly without them.
• volcano-vgpu-device-plugin fork (xiilab/feat/vulkan-vgpu-support): libvgpu submodule bumped to vulkan-v2 build; image rebuilt and pushed as volcano-vgpu-device-plugin:vulkan-v2. (After this Step D PR merges, that image needs another bump to pick up the cherry-picked Tasks 1+2.)
• HAMi parent helm chart upgrade: helm upgrade hami-webhook charts/hami -n hami-system --set global.imageTag=vulkan-v3 --set scheduler.extender.image.tag=vulkan-v3 --set scheduler.admissionWebhook.namespaceSelector.{mode=opt-in,matchLabels={},matchExpressions=[]} --set scheduler.admissionWebhook.objectSelector.{matchLabels={},matchExpressions=[]} --set scheduler.admissionWebhook.whitelistNamespaces=[] --set prometheus.enabled=false (revision 8 deployed).
• Namespace label: kubectl label namespace isaac-launchable hami.io/vgpu=enabled (required by namespaceSelector.mode=opt-in).
• Manifest installer DaemonSet nodeSelector patched to remove legacy hami.io/disabled=true so it actually schedules on GPU nodes.

Verification on ws-node074, isaac-launchable-0 (annotation hami.io/vulkan: "true" + webhook injection)

Path	Method	Result
1 NVML	nvidia-smi --query-gpu=memory.total --format=csv,noheader	23552 MiB (clamped from raw 46068) ✓
2 CUDA	pycuda.driver.mem_get_info()	SKIP (pycuda not installed in image — informational; NVML hook covers the same code path via dlsym)
3 Vulkan memory query	vkGetPhysicalDeviceMemoryProperties device-local heap	24696061952 bytes (23552 MiB) clamp via dispatch ✓
4 Vulkan allocate	vkAllocateMemory(25 GiB)	VK_ERROR_OUT_OF_DEVICE_MEMORY (HAMi-core OOM logged: Device 0 OOM 26843545600 / 24696061952) ✓

HAMI_VK_TRACE lines: 93 (layer fully in chain — Insert instance layer "VK_LAYER_HAMI_vgpu" (/usr/local/vgpu/libvgpu_vk.so)).
runheadless.sh × 3: 3/3 alive exit=124 crash=0.

Sanity

• isaac-launchable-1 (annotation present but pod not yet rolled-restarted): still alive, no manifest volume injected (intended — webhook v3 only injects on new pod creation).
• usd-composer: 3/3 Running, no crash loop.
• All other namespace pods steady.

Production state at end of verification

• /usr/local/vgpu/libvgpu.so: md5 b23793609588d45510d908fa193bfc7b (HAMi-core w/ cherry-picked Tasks 1+2).
• /usr/local/vgpu/libvgpu_vk.so: md5 58ebee3e61dc836d83142a61c51c8139 (Vulkan layer w/ EnumerateDevice* hooks).
• /etc/vulkan/implicit_layer.d/hami.json: 577 bytes, INSTANCE / libvgpu_vk.so / enable_environment + disable_environment.
• hami-vulkan-manifest-installer DaemonSet: 1/1 ready on ws-node074.
• volcano-device-plugin DaemonSet: image vulkan-v2 (containing OLDER libvgpu_vk.so without Tasks 1+2). Follow-up: bump volcano-vgpu-device-plugin libvgpu submodule to current bdd5bbe and rebuild as vulkan-v3 so DS pod restart doesn't reset host .so to the older build.

Spec / Plan

• Spec: docs/superpowers/specs/2026-04-29-step-d-vulkan-opt-in-production-activation.md
• Plan: docs/superpowers/plans/2026-04-29-step-d-vulkan-opt-in-production-activation.md
• Step C redesign spec/plan in same directory.