Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84)

[steve-m]

Summary

An opt-in Vulkan compute backend (make glm VK=1, runtime COLI_VULKAN=1) that runs the GLM decode compute path on any GPU with a Vulkan 1.2 driver — no CUDA, no ROCm. Three tiers, each independently switchable and each falling back to the CPU path on any failure:

• Pinned VRAM expert tier (COLI_VK_EXPERTS, default 320): the top-N routed experts by .coli_usage heat are uploaded once at startup and served from VRAM with no RAM slot, no disk read, no prefetch — issued as one fused async batch per layer (gate+up+silu→down, hidden on-device) that overlaps with the CPU computing the remaining experts. Appears as a new vk bucket in the hit-rate line.
• Dense projections (COLI_VK_DENSE=1): q_a+kv_a fused into one submit, q_b, the o-projection, and the shared expert as a single fused expert-group submit.
• MLA absorb attention core (COLI_VK_ATTN=1): one dispatch per layer covering absorbed query → scores → softmax → weighted latent → value rows, fused with the o-projection so the context vector never leaves the GPU. The latent/rope KV lives in a persistent per-layer device mirror appended at ~2.3 KB/token/layer, with the same invalidation points as the CUDA KV shadow (row rewrite, rebind, resize).

This is a continuation of @kryptt's #84 (the original backend skeleton, tensor-resident coli_vk_matmul, and the int4 nibble-8 shader decode are his design — thank you!). It addresses the two things asked for there: it is rebased onto current dev, and it ships a CPU-reference exactness harness covering every primitive.

Why

1. Hardware reach. Vulkan is the only compute stack that runs everywhere: cards ROCm dropped (RX 580/Polaris via RADV), iGPUs, and boxes where installing a vendor stack isn't an option. The backend needs only libvulkan + a Vulkan 1.2 ICD with subgroup arithmetic, and glslc at build time.
2. It's fast. On an RX 9070 (RDNA4) via Mesa/RADV, the Vulkan expert primitive beats the production ROCm/HIP expert group by ~35%, the attention core beats the HIP kernel by 3.7×, and end-to-end decode is at or above the HIP backend on the same card (numbers below). The llama.cpp folklore that RADV can outrun vendor stacks on AMD holds here once the shaders are properly fused.

Correctness

• Standalone harness (gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk shaders/qmatmul.spv): CPU-reference exactness for every primitive — int4/int8 GEMV across shapes including the long-row o-projection (I=16384), fused gate+up, the full expert group (sync and async issue/take, verified bit-identical to each other), the q_a+kv_a pair, and the absorb attention core including causal S=2, kv_start windows, int8, and long-context (T=2000) cases. Typical maxrel 1e-5…2e-3 (fp32 reduction order on 6144-long dots).
• Engine-level: greedy decode with the full stack enabled matches the pure-CPU engine token-for-token on the validation prompt.
• int4 decodes as offset-binary (nibble−8) with the CPU byte layout — weights upload with a plain memcpy, no repacking.
• Default build: all engine hooks are behind #ifdef COLI_VULKAN; make glm without VK=1 compiles the same code as before (plus three always-false locals). make test-c fully green on the branch.

Measured performance

Hardware/software: AMD RX 9070 16 GB (RDNA4/gfx1201, PCIe 4.0 x16 effective), Ryzen 9 3900X (12C/24T Zen2, OMP_NUM_THREADS=12, threads pinned), 64 GB DDR4-3200 dual-rank, GLM-5.2 744B int4 streamed from NVMe (~19 MB/expert). Mesa/RADV 26.1.4, Vulkan 1.4 ICD; ROCm 7.2.4 + HIP for the comparison backend. Model: 78 layers, 256 experts/layer, kv_lora 512, MLA/DSA.

Primitive microbenchmarks (per-call including readback, K-expert int4 MLP 6144→2048→6144):

Primitive	Vulkan (RADV)	ROCm/HIP production
Expert group, K=8	0.117 ms/expert	0.179 ms/expert
Expert group, K=32	0.107 ms/expert	0.179 ms/expert
Decode attention core (t_acore, 16 tok × 78 layers)	0.61 s	2.2 s (HIP kernel: 5.7 s CPU: n/a)

End-to-end GLM-5.2 decode (same box, same engine settings, expert-routing top-p 0.7, drafts off, interleaved/batched A/B runs; our fork additionally carries an unrelated dual-SSD-mirror patch on the same base — active for both backends in every comparison, so the deltas are backend-attributable):

Generation length	Vulkan	ROCm/HIP
64 tokens	1.74–1.78 tok/s	1.53–1.56 tok/s
256 tokens	1.53–1.67 tok/s	1.51–1.53 tok/s
512 tokens (sustained, 5.4 min)	1.58 tok/s	—
Prefill (11–23 tokens)	7.2–8.5 s	8.1–12.4 s

Run-to-run variance on this box is ±0.1 tok/s; the 64-token advantage (~+13%) narrows to ~+5% at long form as the attention window and expert-I/O share grow for both backends.

The two memory-type rules that made it work (documented in docs/vulkan.md, learned the hard way):

1. Buffers the CPU reads back must be HOST_CACHED — reading write-combined ReBAR VRAM from the CPU runs at ~40 MB/s and was a hidden 5.6× per-call penalty.
2. Everything else (weights, inputs, the KV mirror) lives HOST_VISIBLE|DEVICE_LOCAL, so uploads are plain memcpys and the GPU reads at VRAM speed.

Integration surface

• New: c/backend_vulkan.c (~1.3k lines, plain C99 + libvulkan), c/backend_vulkan.h, c/shaders/qmatmul.comp, c/shaders/qmatmul_gate_up.comp, c/shaders/attention_absorb.comp, docs/vulkan.md.
• Modified: c/glm.c (hooks under #ifdef COLI_VULKAN), c/Makefile (VK=1 block + glslc rule), docs/ENVIRONMENT.md, README.md.
• Env: COLI_VULKAN, COLI_VK_SHADERS, COLI_VK_EXPERTS, COLI_VK_DENSE, COLI_VK_ATTN.
• 18 bisectable commits, each building; the history walks port → shader fusion → integration → attention core → tier.

One operational note worth flagging beyond this PR: the engine's OMP self-tune (OMP_WAIT_POLICY=active + spin) is skipped under COLI_CUDA/COLI_METAL but not under other configurations. With 12 pinned spinning threads the async I/O pool starves (measured 28 → 5 GB/s CPU expert bandwidth, ~2.8× end-to-end). docs/vulkan.md tells Vulkan users to set COLI_NO_OMP_TUNE=1; extending the self-tune skip to any GPU backend (or to PIPE=1 runs) may be worth a separate look.

Limits / future work

• Decode-focused: the expert tier and attention core serve S≤4; prefill keeps the CPU/batched paths (dense projections do run on VK at prefill, which is why prefill got faster).
• DSA top-k selection, ragged multi-slot serving, and quantized-KV caches fall back to the CPU attention path (same guards as the CUDA absorb).
• Validated on RDNA4/RADV. The shaders use dynamic subgroup sizes (wave32/64-safe) and Vulkan 1.2 only, so Polaris/gfx803 and other vendors should work — testers welcome, especially RX 580 owners.
• Ready as follow-ups, deliberately not in this PR: an fp8 KV mirror for KV cache quantization: fp8 (KV8) + 4-bit TurboQuant (KV_TQ) on CPU, CUDA, and Metal #399's KV8=1 (already implemented and validated against kv_fp8.h on our side — the absorb shader decodes e4m3 in place, 4× less mirror traffic; this PR just cleanly falls back to CPU attention when the f32 cache is absent), cooperative-matrix prefill kernels, and a fully resident-layer pipeline.

How to run

cd c && make glm VK=1
COLI_VULKAN=1 COLI_VK_DENSE=1 COLI_VK_ATTN=1 \
PIN=<model>/.coli_usage PIN_GB=0 COLI_NO_OMP_TUNE=1 \
./coli run "Hello" --topp 0.7

[JustVugg]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

[steve-m]

Rebased onto current dev — the split turned out painless: git tracked glm.c → colibri.c as a rename, so all the engine hooks followed automatically and only the Makefile hunk needed hand-placement (the VK=1 block now hangs off the colibri rule; VK= was added to the build-config stamp). Re-validated after the rebase: default build + make test-c green, the -DVK_TEST harness passes on the RX 9070, and an end-to-end run with the full stack reproduces the reference output. #421 got the same rebase since it also touched the old glm.c. Thanks for the heads-up and the offer — no thorny hunks this time.

[steve-m]

pp512/tg128-style numbers, for comparability with the llama.cpp benchmark conventions (same box and config as the tables above; dual-SSD mirror active for both backends; fresh process per run; two runs each):

Workload	Vulkan (RADV)	ROCm/HIP
pp526 — 526-token prompt (target 512)	5.01 / 5.11 tok/s	4.50 / 4.47 tok/s
tg128	1.53 / 1.55 tok/s	1.47 / 1.49 tok/s

Notes: prefill here is expert-streaming-bound (a 512-token batch-union touches nearly every expert of the 744B model), so pp measures the whole engine + disks more than the GPU. The accounting slightly favors HIP — Vulkan uploads its dense weights during the first prefill while the CUDA/HIP tier uploads at load, before the prefill timer — and Vulkan leads regardless (+13% pp, +4–5% tg). These standard workloads also reproduce much more tightly than free-running generations (≤2% spread), so we'll use them for future comparisons.

[JustVugg]

Rebase needed: #298 just merged into dev and reworked the quantized-kernel scale handling (fmt=4 per-group scales) — your Vulkan kernels will want the same semantics or g64 containers will produce garbage on Vulkan, same bug #298 fixed on CUDA. The conflict is semantic and the backend is yours end-to-end, so it needs your hands; no rush, but flag if you've stopped working on it so we know where it stands.

[steve-m]

Rebased onto current dev (cbbe310, post-#298/#421/#192) and implemented the #298 semantics you flagged — plus the format work that landed in dev since the last push made a bigger scope natural.

Grouped scales in the Vulkan kernels (the #298 ask). All three kernels (dense GEMV, fused gate+up, absorb attention) now decode fmt=4 grouped int4 with one scale per gs inputs, applied to the packed-word partial; per-row scaling is skipped exactly as #298 did on CUDA. The group size travels as an explicit parameter into the resident tensor and the push constants. The host gates fmt=4 to gs % 8 == 0 (a packed word never straddles a group) — anything else falls back to the CPU path #298 already fixed. And since dev now carries #168, the same treatment covers fmt=5 int3-g64 (fixed g64 groups, two-plane decode) across all three kernels — a full GLM int3 container runs the complete Vulkan stack.

Validation. The -DVK_TEST harness gained fmt=4 cases (gs=64 across the real GLM shapes — dense, o-proj I=16384, S=8 batch, expert_group, matmul_pair, absorb incl. S=2 causal + kv_start window — plus a gs=32 sanity case) and fmt=5 cases; full suite passes on the RX 9070 at maxrel ≤ 5e-4 (absorb ≤ 2e-5). End-to-end greedy runs reproduce the reference output on both a per-row int4 container and an int3-g64 container (step-1 top-5 logits identical to the CPU path).

Also in the refresh, from production use on our box:

• VRAM pressure-proofing: allocations carry VK_EXT_memory_priority classes (scratches/KV mirror 1.0 > dense 0.75 > expert tier 0.4) and the tier fill stops against VK_EXT_memory_budget while COLI_VK_RESERVE_GB (default 3) stays free for the lazily-allocated dense + KV working set, with an explicit log line. Both are runtime-detected no-ops on drivers without the extensions.
• Arena suballocation: weight tensors now share 256 MB device-memory blocks instead of two vkAllocateMemory calls per tensor (a large tier was ~5.7k allocations — also uncomfortably close to maxMemoryAllocationCount on some drivers).
• Readiness-ordered CPU share: the expert block computes pipe-ready/resident experts first so a still-loading expert's I/O hides behind their matmuls (ordering hint only; every entry still waits properly). t_ecpu now counts kernel time only, waits go to t_ewait.
• PROF additions: a VK-BLOCK line (per-block anatomy: classify/issue/take + avg nvk/ncpu) and VK_PROF=1 submit-vs-fence-wait totals across all sync paths.

One measured caveat worth documenting: on our RX 9070/RADV, growing the resident expert tier past ~6 GB slows the other GPU paths' per-dispatch execution (~1.3 s per tier-GB over a 64-token decode; profiled to the fence wait, not the submit call, with eviction/allocation-count/free-memory all ruled out) — so the sweep-derived COLI_VK_EXPERTS=320 default stands and the budget stop is a guard, not an invitation to fill the card. Details in the docs.

Numbers on the box are unchanged from the tables above (the refresh is format coverage + hygiene, not a perf change): 64-tok tg on the int3-g64 container now reaches 1.73–1.75 tok/s with the engine-side improvements that landed in dev (#421 mirror + the .qs fold-in we'll propose separately).

[JustVugg]

Thanks @steve-m — this is a big, welcome piece of work, and I want to get it in. Two things before merge, one mechanical and one that only you can provide (I don't have a Vulkan GPU to verify it myself).

1. CI is red for a mechanical reason — 9 committed test binaries. The PR added compiled binaries under c/tests/ (test_dsa_select, test_i4_grouped, test_kv_alloc, test_kv_disk, test_kv_fp8, test_logit_nan, test_sample_nan, test_stops, test_topp). On a fresh macOS checkout make sees them as up-to-date and skips rebuilding, then run_tests.py hits OSError: [Errno 8] Exec format error trying to exec a Linux ELF on macOS — that's the make check failure. (Same class of thing that hit #421.) Please git rm those 9 files and add them to .gitignore; the .c sources stay. That alone should turn CI green.

2. Because it's a new hardware backend I can't run, I'd like to read validation data before merging — this is the bar we hold every backend to. Could you post, from your Vulkan GPU:

• Correctness first: confirmation the Vulkan path is coherent, ideally a teacher-forcing TF=1 token-exact check vs the CPU oracle on a tiny model (the same 32/32 gate the CPU/CUDA paths pass), or at minimum a real chat transcript that's clearly coherent. For a new backend, correct output matters more than speed — cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 was a CUDA path that looked right and produced garbage.
• Then perf: decode + prefill tok/s with the Vulkan tier vs CPU-only on the same box/prompt, expert hit rate, and which GPU / driver / Vulkan version.

Plan: once CI is green (step 1) and the data checks out (coherent output + tok/s in step 2), I'll review and merge. I've verified the mechanical side is the only CI blocker; the rest is your GPU's numbers. Happy to fix the committed-binary bit for you and push to your branch if you'd rather — just say the word.

[steve-m]

Both items addressed — plus one more commit that was in flight.

1. CI / committed binaries: removed from the entire history, not just the tip. All nine entered through one commit (an over-broad git add on my side — apologies); the branch is rewritten so no commit in the PR ever contains them, verified per-commit across the range, and c/.gitignore now covers the full current test-binary set (including test_int3, test_tok_o200k, test_pipe_block, which weren't in your list yet) so the class is closed. Force-pushed; a fresh checkout of any commit builds its test binaries from source.

2. Validation from the Vulkan GPU. Hardware: AMD Radeon RX 9070 (RADV GFX1201), Mesa 26.1.4, Vulkan 1.4.354, Linux 6.19. Model: GLM-5.2 744B, per-row int4 container, dual-SSD mirror (#421), full Vulkan stack = dense + absorb attention + 320-expert VRAM tier.

Correctness. The tiny-model TF oracle fixture isn't bundled, so here is the equivalent evidence on the real model:

• Coherent transcript (sampled, topp 0.7, 96 tokens, full VK stack): "# The Keeper of Tidesmoor Light — The foghorn had been malfunctioning again. Eliana pulled her coat tighter around her shoulders and squinted into the gray morning, the lighthouse door whistling shut behind her. The sea churned below the cliffs—a restless, churning darkness of mercury and green-black water…" — clean prose throughout.
• Greedy factual agreement: CPU and Vulkan both answer The capital of France is → "Paris." (continuations diverge after the answer — see the honesty note below).
• Primitive-level exactness is where the rigor lives: the -DVK_TEST harness checks every kernel against a CPU reference (int8/int4/fmt=4/fmt=5, real GLM shapes) at maxrel ≤ 5e-4, and a built-in engine verify mode (COLI_VK_QPREP=2) compares the fused attention chain against the split path per layer on the real 744B weights: all 78 layers within 1.8e-4, kv path bit-exact.
• Honesty note on token-exactness: on this 744B int4 container, any fp-ordering change shifts deep-layer logits O(1) through MoE-router flips — pure-CPU vs CPU-with-different-kernels diverges in continuation exactly like CPU-vs-Vulkan does, with the factual argmax stable. Token-exact TF across backends is achievable on a small well-conditioned model (happy to run the 32/32 gate if you can point me at the glm_tiny generator), but not on this container for any backend pair.

Performance — standard workloads (pp526 = 526-token prompt, tg128; fresh process per run, two runs each, same box/prompt/config, only COLI_VULKAN toggled):

Workload	Vulkan (full stack)	CPU-only
pp526	4.90 / 4.92 tok/s	4.06 / 4.05 tok/s
tg128	1.48 / 1.52 tok/s	1.12 / 1.13 tok/s

+21% prefill, +32–35% decode. Expert hit rate ~80% both (VK serves 15.2–15.5% of lookups from the VRAM tier with zero disk I/O; CPU-only covers the same hit rate from RAM alone — the delta is the GPU offload, not cache coverage).

3. New commit since the refresh: single-submit q-prep chain (rmsnorm.comp + coli_vk_attn_qprep): [q_a + kv_a] → on-GPU RMS-norm → q_b recorded under one fence instead of three per layer (the middle roundtrip existed only for the CPU-side norm; the normed latent still returns to the host for the DSA indexer). Sync submits per 64-token decode drop 14.3k → 9.3k, measured −1.0 s of GPU wait; wall-clock neutral on this I/O-bound box today, and it comes with a kill-switch (COLI_VK_QPREP=0) and the per-layer verify mode above. Validated the same way as the rest: harness cases (S=1/2/11) + the 78-layer real-weight comparison.

[steve-m]

Rebased onto current dev (dce7012, v1.1.0) — clean replay, one trivial
TEST_BINS union in the Makefile. With #500 merged the stray-binary CI
failure is gone, so this PR's CI should finally run green; happy to have it
re-triggered.

Two new commits since the last push, both from profiling decode on the
RX 9070 box:

vulkan: preload the dense working set before the tier fill — the dense
projections (q_a/q_b/kv_a/kv_b/o, shared expert) uploaded lazily on first use,
i.e. AFTER the expert tier had already filled to the VRAM budget. On a
VRAM-tight box the dense working set then spills to GTT: we measured 2.1 GB
spilled and +34% on every absorb-attention call. Preloading dense right before
the tier fill lets the tier self-size through the existing budget stop instead
(spill 2.1 → 0.2 GB, tier sheds only as many experts as the dense set needs).

vulkan: spin-then-block fence waits on the sync paths — decode
dispatches are microsecond-scale, so a blocking vkWaitForFences pays wakeup
latency on every roundtrip. All sync-path waits now poll vkGetFenceStatus
for up to COLI_VK_SPIN_US (default 300 µs, 0 reverts to blocking) before
falling back to the blocking wait.

Standard workloads, same box/prompt/config/convention as the tables above
(pp526 + tg128, fresh process per run, two runs each; this branch build):

Workload	previous push (posted above)	this push
pp526	4.90 / 4.92 tok/s	5.13 / 5.15 tok/s
tg128	1.48 / 1.52 tok/s	1.74 / 1.81 tok/s

Decode attention dropped 26.6 → 15.8 s per tg128 run; what remains of decode
is dominated by expert disk I/O (felt wait 21-24 s — also the source of the
r1/r2 spread), which is outside this PR's scope. Disclosure: part of the tg
delta comes from pinning the GPU's DPM performance level
(power_dpm_force_performance_level=high) — microsecond-burst duty cycles
never ramp RDNA4's memory clock on their own, worth knowing for anyone
benchmarking this backend; the dense-preload/spin-wait commits are the rest.
A nice side-effect visible in these runs: with the desktop compositor holding
VRAM, the tier now self-sizes below its cap (320 → 296 in r1) through the
existing budget stop with zero GTT spill — exactly the failure mode the
preload commit closes.

Validation on this exact branch build (RX 9070, RADV, Mesa 26.1.4): VK=1
clean build, the CPU-ref exactness harness full PASS (absorb fmt 1/2/4/5
incl. the fused absorb+project paths, maxrel ≤ 1.6e-5), engine smoke coherent.

Follow-up we're preparing as a separate PR: a second-device Vulkan expert
tier (COLI_VK_DEV2) that fills another GPU's VRAM with the next-hottest
experts — an RX 580 beside the 9070 takes us past 2 tok/s decode on the same
box. Kept out of this PR to keep the review surface stable.

[noobdev-ph]

Tested this on RDNA4 (RX 9070 XT / gfx1201, RADV, Mesa 26.1.3) since @JustVugg mentioned not having a Vulkan GPU to verify against — it came up and ran correctly first try on a box it had never seen. Nice work.

Two things that should be useful for the merge:

• The cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 grouped-scale semantics check out on a real g64 container. Ours is a genuine fmt=4 / gs=64 conversion of the full 744B checkpoint, and output was clean and on-topic in 50/50 runs, byte-identical across every configuration tested — so the per-group scales look correct on RDNA4 against real quantized weights, not just the reference harness.
• Resizable BAR turns out to be required on discrete cards, and its absence fails silently. With ReBAR off, HOST_VISIBLE|DEVICE_LOCAL only exists in a 256 MB window, so the tier lands in system RAM while still logging 320 hot experts resident (6.04 GB VRAM) — the card sat at 82 MB and it ran slower than the CPU path. Enabling ReBAR took it from 0.11 → 0.24 tok/s. A one-line init warning comparing the heap size against the host-visible+device-local type would save people a very confusing first run.

Also a small one: COLI_VK_SHADERS unset falls back to "shaders/qmatmul.spv" relative to CWD (colibri.c:6955), so it only resolves when launched from c/ — and it wants the full path to the .spv file, not the shader directory.

Full numbers, methodology and a ROCm/HIP comparison are in #523 rather than here, to keep this thread focused. There's an open question there about the ~35% faster than ROCm on RDNA4 note in c/Makefile:260 — I measured the opposite on this box and am fairly sure I've configured something differently, so I've documented my whole setup and listed what I'd happily re-run. The card and checkpoint are set up here, so just say what would be most useful.

[steve-m]

Rebased onto current dev (e4b5bf3, v1.1.1 + the Inkling merge) — clean replay, diff scope still VK-only. Three commits on top, from the field reports in #523 and this thread:

• ReBAR init warning — the discrete-card silent-GTT-fallback @noobdev-ph and @JustVugg both flagged: init now compares the chosen memory type's heap against the device-local heap and warns with the firmware fix named. docs/vulkan.md states the requirement.
• Shader-path resolution — COLI_VK_SHADERS accepts a directory or the .spv; unset, resolves next to the binary; the error names what it tried.
• PREFETCH=1 residency skip — completes the vk_reg_served guard across the one prefetch path that lacked it (off by default).

Also reworded the ~35% faster than ROCm note in c/Makefile/README.md to state what it measured (the VRAM-resident expert-group primitive, per-call) rather than implying end-to-end — @noobdev-ph correctly measured the opposite e2e on a storage-bound box.

On the substantive #523 items: the DRAFT confound you identified is exactly it, and I posted the matched DRAFT=0 comparison from our box (both backends) plus the MTP-acceptance data over there — short version, VK comes out ~13% ahead full-stack once speculation is off on both arms, and acceptance under the VK tier doesn't collapse, so I don't think the #163 guard needs to extend to Vulkan. Detail + numbers in #523 to keep this thread focused. CI is green.

[steve-m]

Rebased onto current dev (21e7a35) — clean replay, diff scope unchanged (13 files, VK-only). No functional changes since the last push; this is purely the rebase that clears the merge conflict.

One thing worth noting from it: dev picked up the fmt=6 E8/IQ3 container (#465, #501) in the meantime. The Vulkan paths are unaffected by construction — every VK entry point is gated on the VK_FMT_OK allowlist (fmt 1/2/5 plus grouped int4), so fmt=6 tensors stay on the CPU path instead of reaching a shader that can't decode them. A VK fmt=6 kernel would be separate work.

Validation on this exact branch build (RX 9070, RADV, Mesa 26.1.4):

• VK=1 clean build; all four shaders compile to SPIR-V under --target-env=vulkan1.2
• CPU-ref exactness harness full PASS — fmt 1/2/4/5, the fused gate+up pair, and the absorb / absorb+project paths
• make test green: 197 Python tests (13 skipped) plus the C suite
• Engine smoke on a real GLM int4 container, coherent output with the VK tier serving

[InspiringCode]

Thank you a lot for your hard work! It's so exciting to see all the progress on local AI inference! 😃

Does this also help with iGPUs like the AI MAX 395? What's the correct/best setup for such a box? Set GPU RAM to max possible (e.g. 96 GB)?

[steve-m]

Rebased onto current dev (b3fafb1) — clean replay, no conflicts. On top of the rebase this adds fmt=6 (E8/IQ3) support to the Vulkan paths.

That is a change of position from my last comment, where I said the VK paths were unaffected by the E8 container "by construction" because every entry point is gated on VK_FMT_OK. That was true, and it was also the problem: with #627 giving CUDA an fmt=6 kernel, Vulkan became the remaining backend where an E8 container silently pushed all of its routed experts onto the CPU tier — the exact inversion of what the format is for.

What it does

Decode in qmatmul.comp and qmatmul_gate_up.comp, striding 32-weight sub-blocks (the granularity at which the sign/scale word changes, so the two grid rows a lane touches stay hot across its four 8-weight lanes). The awkward part of the layout is that every field inside a 98-byte super-block is 4-aligned relative to the block start, but 98 % 4 == 2, so blocks alternate 4- and 2-aligned within a row — hence byte-wise word assembly, with the straddle shift only ever 0 or 16. The fp16 super-scale is read 16 bits wide on purpose: its 32-bit window can reach past the last row of the slab, its two bytes never do.

The rotation, which is what the expert tier actually needs. fmt=6 stores W@Q, and the split follows the CPU and CUDA paths exactly: all routed experts of a layer share the gate/up input, so the caller rotates it once per layer, while the down projection's input is the per-expert silu product — produced and consumed on the GPU. Sending it back to the host would undo the point of the fused expert group, so e8_rot.comp rotates it in place between the two phases of the same submit, over the whole hidden buffer at once (the transform is per row, so every expert's slice rides one set of dispatches instead of paying a per-expert cost). Blocks tile block-diagonally exactly as e8_rot_rows does.

Scope is narrower than it might look: --ebits/--io-bits are integer bit widths and only --xbits/--{up,gate,down}-bits accept e8, so only routed expert projections can ever be fmt=6. Attention and dense tensors cannot reach the format, and VK_FMT_OK is left alone. The converter also refuses any input dim that is not a whole number of 256-weight blocks, which upload_tensor re-checks, so the shader has no ragged tail to bounds-test.

moe() fed x straight to the tier at three sites inside the VK block; all three now take the rotated copy. The registry serves an expert with no RAM slot at all, so its format is only knowable from the resident tensor — hence coli_vk_tensor_fmt. The block's two CPU fallbacks (the CPU share and the device-lost recompute) were also missing the down-input rotation the main CPU path does — unreachable before, since fmt=6 could not be uploaded at all, but live the moment this lands.

If shaders/e8_rot.spv is absent, coli_vk_e8_ok() reports 0 and fmt=6 tensors stay on the CPU rather than decoding against an unrotated input.

Validation

New make vk-test (tests/test_vk_e8.c; on demand, since it needs a device) checks the shader decode against quant.h's own matmul_e8, and the full expert MLP — including the device-side rotation — against the CPU path, with int4 held on one side to isolate the fused gate_up branch from the rotation:

fmt=6 decode vs quant.h matmul_e8:
  matmul S=1 I=6144 O=1536       rel_l2=1.6e-07   ok
  matmul S=1 I=16384 O=512       rel_l2=2.1e-07   ok    (unstaged path)
expert MLP (gu=gate/up fmt, dn=down fmt):
  gu=2 dn=2 n=2 D=1024 I=512     rel_l2=9.79e-07  ok    (int4 baseline = the noise floor)
  gu=6 dn=2 n=2 D=1024 I=512     rel_l2=3.26e-07  ok    (fused gate_up E8, no rotation)
  gu=2 dn=6 n=2 D=1024 I=512     rel_l2=7.58e-07  ok    (device rotation + E8 down)
  gu=6 dn=6 n=2 D=6144 I=1536    rel_l2=2.87e-07  ok

Errors are relative L2 on purpose — per-element relative error is the wrong metric for these long, sign-cancelling dot products, where individual outputs land near zero by cancellation and their relative error explodes on summation order alone. fmt=6 sits at or below the int4 baseline. Green on RADV GFX1201 (RX 9070) and llvmpipe, so both subgroup widths. make check 220.

shaders/e8_grid.glsl carries a second copy of quant.h's 256×4 codebook, because GLSL cannot include the C header — the same arrangement tools/iq3xxs_grid.json already has for the Python codec. make vk-test is what keeps the two honest.

Measured

GLM-5.2 (744B, E8 container converted with --group-size 64 --ebits 4 --xbits e8), RX 9070, 32-token greedy decode. Dense projections and the absorb attention core are on the GPU in both rows, so this isolates the expert tier:

	decode	expert hit rate
without the fmt=6 expert tier	24.3 s (1.32 tok/s)	66.0% (all LRU)
with the fmt=6 expert tier	19.6 s (1.63 tok/s)	69.6% (LRU 37.7% + vk 31.9%)

+24%, generated text character-identical between the two. 440 experts resident in 6.36 GB = 14.45 MB/expert, exactly 3 × (98 B per 256 weights) — about 14% more experts per GB of VRAM than int3-g64, which is the capacity argument for the format.

Two caveats so these are not read as more than they are. They come from a container whose CPU-side matmul_e8 I also had to vectorise — upstream ships only the scalar reference kernel, which was 92% of decode time on this model — and that work is not in this PR; it is a CPU kernel and is now #654. The converter-side work that produced the container is #655. And the run is cold, on a 32-token sample, with .coli_usage seeded from a single short run; a warm run improves both rows.

One thing worth flagging to maintainers

c/shaders/qmatmul.comp is shared between engines, and #602 also uses p.fmt == 6 in it — for grouped-asymmetric int4 (GPTQ), not the E8 lattice. Two different formats, the same push-constant value, the same shader file. Whichever of the two merges second will silently break the other's decode, with no build error to catch it. The container-level number is already fixed at 6 for E8 in quant.h and on disk, so the Qwen engine's engine-local numbering looks like the side that should move — but it wants a decision either way rather than being discovered at merge time.

[steve-m]

Rebased onto current dev (v1.3.0, post-#676) — conflict-free and green again. And since Kimi K3 is now upstream, this PR gains its Vulkan support, following the same pattern as the fmt=4/fmt=5 kernels already here:

• fmt=7 (MXFP4) decode in qmatmul.comp: e2m1 nibble LUT + one scale per 32-input group, reusing the cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 grouped-scale path. The host expands the ue8m0 exponents to f32 at upload, so the QAT expert bytes are uploaded exactly as stored — never re-encoded. Validated against the CPU matmul_mxfp4 on K3 expert dims: rel_l2 2.2e-07 on an RX 9070 (RADV GFX1201), 2.6e-07 on llvmpipe (tests/test_vk_mxfp4.c; the test skips cleanly without a Vulkan device).
• kimi_k3 Vulkan tier (make VK=1 kimi_k3; K3_VK=0 runtime opt-out): shared experts VRAM-resident at init via the existing fmt-1/4 shaders (all 276 matrices = 7.5 GB on a 16 GB card), plus a fill-once routed-expert tier in fmt=7 — experts enter from freshly-read RAM slots (K3_VK_UP/step, budget K3_VK_GB), and tier-resident experts then skip both the 17.5 MB disk read and the CPU matmuls at decode. CPU fallback on every path; output identical.

Full-model behaviour on the 2.8T snapshot (62 GiB box, RX 9070): the tier registered 13 GPU hits within the first 8 steps of filling (~1.6× random — K3's short-term routing locality is real, despite its Quantile-Balancing-flat marginals), decode at parity with CPU while the tier is cold, with the I/O savings accruing as a resident server fills the remaining ~9.5 GB of budget. Docs updated (docs/kimi_k3.md: tier section + env table).

[steve-m]

Note on the comment above: the merge and my push crossed by a few seconds — the two Kimi commits (fmt=7 MXFP4 + the kimi_k3 VK tier) landed on the branch just after the merge point, so they are not in the merged state. They're now #705, applying directly on top of this merge. Thanks for taking the backend!