Vulkan: fmt=7 MXFP4 decode + Kimi K3 GPU tier (follow-up to #418) - #705
Merged
Conversation
…imi K3 experts New fmt=7 branch in qmatmul.comp: e2m1 LUT nibble decode (bit3 = sign, low nibble = even column — same packing as the K3 checkpoint), one f32 scale per 32-input group reusing the JustVugg#298 grouped-scale path; the host pre-expands the ue8m0 exponents to f32 at upload (mx4_scale), so the shader stays float-only. Host gates in tensor upload/scale sizing accept fmt=7 with word-aligned groups. Validated against quant.h matmul_mxfp4 on K3 expert dims (I=3584, O=3072, S=2, random nibbles + exponents 2^-7..2^4): rel_l2 2.6e-07 (tests/test_vk_mxfp4.c, skips without a Vulkan device). (cherry picked from commit ca5a1b254d8714361910f7d5eabaa072a0bdf11f)
…FP4 routed-expert tier Build with `make VK=1 kimi_k3` (backend_vulkan.o + shaders; default CPU build untouched). Two residency classes on the card, both with transparent CPU fallback and identical output: - dense W tensors: shared experts uploaded at init (they run every token and are the largest always-on RAM-bandwidth slice that fits VRAM); any uploaded W computes on the existing fmt=1/4 shaders at S==1 via the w_matmul hook. - routed experts: a fill-once tier in fmt=7 (native MXFP4 bytes straight from the RAM slot, ue8m0 exponents expanded to f32 group scales at upload; QAT weights never re-encoded). Experts enter from freshly-read slots on the main thread, K3_VK_UP per step (default 8), until the VRAM budget (K3_VK_GB, default driver budget) is full. At decode (C==1) tier-resident experts drop out of the disk union — skipping BOTH the 17.5 MB read and the CPU matmuls — and run as one paired w1/w3 submit + SiTU-GLU on CPU + w2 down. Chunked prefill stays on the CPU-batched path (and still warms the tier through the upload hook). K3_VK=0 disables at runtime; [K3-VK] stderr lines report residency and GPU hits (also after every serve-mode request). (cherry picked from commit 3bcdfd0974eef2e1953a6a849b6c785c31e93d79)
monotophic
added a commit
to monotophic/colibri
that referenced
this pull request
Jul 30, 2026
While this PR sat open holding the maintainer-assigned ordinal 7 (JustVugg#524), dev merged JustVugg#705: MXFP4 for the Kimi K3 Vulkan tier, tagged fmt=7 (backend_vulkan.c). Two live formats, one number -- and this format already lost an ordinal once before, when JustVugg#465 claimed its original fmt=6 upstream (the incident colibri.c's PRIVATE ORDINAL BLOCK comment records). Same weakness, second bite: ordinals are only settled by merge into dev, and nothing forces an open PR's claim to stay visible. Pure retag, zero on-disk impact: qt_resolve_fmt infers format from byte arithmetic alone and no container carries the ordinal, so fmt=8's semantics are bit-identical to what this branch shipped as fmt=7 (same raw-e4m3 + per-128x128-block-f32 layout, same inverted collision policy, same explicit allow-list gating). Every reference across the engine, Metal path, tests, tools, and docs moves 7 -> 8; dev's fmt=7 (MXFP4) references are untouched; the renumber-narrative comments (colibri.c QT block, quant.h, repack tool docstring, test headers) are updated to record this second renumbering factually. Fix round 1 (review findings folded in, keeping this commit's completeness claim true): check_wire_split's two fp8 call sites passed fmt=7 while only their labels were renumbered -- with 7 no longer a QT format, both sides of the assertion fell to the same per-row default and the checks were tautologies. Call sites now pass 8, and the helper gained (a) a supported-set self-guard on its fmt argument, so label/argument drift dies loudly at the call site, and (b) an fmt==8 independent block-grid pin mirroring the existing fmt==6 literal pin -- want and got both flow through the engine's own qt_scale_bytes(), so only an independent recomputation makes a reverted fmt==8 branch actually fail the test (mutation-proven). Also caught by the same review: the current-contract fmt list in backend_metal.h still said 7=fp8-e4m3, one stale "not to include 7" comment in test_backend_metal.mm, and two stale fmt=4/5/7-era cross-reference comments. Surviving fmt=7 mentions in this branch's files are exactly two categories: references to dev's MXFP4 (which really is fmt=7) and historical narrative prose about the ordinal this format used to hold.
monotophic
added a commit
to monotophic/colibri
that referenced
this pull request
Jul 30, 2026
…ssignment Registry follow-through for the renumber commit on PR 1's branch: our fp8-e4m3-b128 row moves to fmt=8, and fmt=7 gains its real, factual row -- MXFP4 (e2m1 nibbles + per-32-group f32 scales), Kimi K3's native routed-expert tier, Vulkan backend only, merged via JustVugg#676/JustVugg#705. A registry whose rows disagree with dev at birth would defeat the document's purpose. New "ID assignment" paragraph states the rule the last two weeks demonstrated twice: an ordinal is claimed by the first merge into dev; there is no reservation mechanism; an assignment on an open PR or issue does not survive a competing merge; proposers must scan dev AND open PRs; this registry is the coordination point. Also re-verified every c/colibri.c / c/quant.h line number cited in the Sources section against the current restack (base dev 292ed4c) and updated the verification paragraph accordingly, plus the one stray fmt=7-referencing stamp comment in tests/test_int3.c.
This was referenced Jul 30, 2026
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The #418 merge and my last push crossed by seconds — these are the two commits described in my comment there that just missed it. They apply directly on the merged backend (their base is #418's merge point); everything below is validated on real hardware and weights.
qmatmul.comp: e2m1 nibble LUT + one scale per 32-input group, reusing the cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 grouped-scale path. The host expands the ue8m0 exponents to f32 at upload, so the QAT expert bytes are uploaded exactly as stored — never re-encoded. Validated against the CPUmatmul_mxfp4on K3 expert dims: rel_l2 2.2e-07 on an RX 9070 (RADV GFX1201), 2.6e-07 on llvmpipe (tests/test_vk_mxfp4.c; skips cleanly without a Vulkan device).kimi_k3Vulkan tier (make VK=1 kimi_k3;K3_VK=0runtime opt-out): shared experts VRAM-resident at init via the existing fmt-1/4 shaders (all 276 matrices = 7.5 GB on a 16 GB card), plus a fill-once routed-expert tier in fmt=7 — experts enter from freshly-read RAM slots (K3_VK_UP/step, budgetK3_VK_GB), and tier-resident experts then skip both the 17.5 MB disk read and the CPU matmuls at decode. CPU fallback on every path; output identical.Full-model behaviour on the 2.8T snapshot (62 GiB box, RX 9070): the tier registered 13 GPU hits within the first 8 steps of filling (~1.6× random — K3's short-term routing locality is real despite its Quantile-Balancing-flat marginals), decode at parity with CPU while the tier is cold, with the I/O savings accruing as a resident server fills the remaining ~9.5 GB of budget. Docs updated (
docs/kimi_k3.md: tier section + env table).make checkgreen; the default CPU build is untouched.