Add DeepSeek V4 Flash CPU inference with NVMe expert streaming

[DrewZt]

Summary

This PR adds a separate, model-specific DeepSeek V4 Flash + DSpark CPU engine for x86-64 Linux and Windows/MSYS2. It streams routed FP4 experts directly from the original safetensors checkpoint on NVMe and allocates available memory between resident target tensors, DSpark stages, the output head, and per-layer expert caches.

The implementation includes:

• native C inference for DeepSeek V4 Flash;
• direct FP8 dense-tensor and native FP4 routed-expert loading;
• NVMe expert streaming with caching, prefetching, and hot-expert pinning;
• sparse attention and compressed KV state;
• DSpark speculative drafting and verification;
• automatic RAM-tier planning;
• a dedicated make deepseek-v4 target and standard-library-only c/v4 launcher;
• unit, ownership, malformed-input, and optional full-model validation.

The DeepSeek implementation remains a separate model-specific engine and does not modify the main c/colibri.c inference or storage paths.; the default GLM build and unsupported-platform test path remain intact.

Merge hardening

This branch was reconstructed as a fresh forward port onto current upstream dev, rather than replaying the obsolete pre-#391 commit history. and integrates V4 targets with the upstream target-triplet detection, tools/run_tests.py, and tools/clean.py flow.

The safetensors index now:

• frees every parsed JSON tree on success and failure;
• rejects headers larger than 512 MiB or outside the shard;
• validates root, metadata, dtype, offsets, and shape JSON types before dereferencing;
• rejects negative, fractional, NaN/Infinity, and out-of-range numbers;
• checks shape-product and file-offset arithmetic for overflow;
• requires payload bytes to equal dtype width ? shape numel, preventing undersized reads and oversized writes.

Malformed fixtures cover bad header lengths, wrong JSON types, invalid offsets, invalid dimensions/rank, arithmetic overflow, dtype/shape/payload-size mismatches, and repeated open/close under LSan. The same JSON-tree ownership fix was applied to V4 config parsing after sanitizer validation exposed it.

V4 config integer fields and compress_ratios now require finite, integral values in the C int range. Floating-point fields require finite values representable as float.

The full-model oracle launcher resolves its local binary path correctly and requires exact output length and token identity for both DSpark-on and DSpark-off runs. The resident dense cache always uses the engine-owned canonical config and no longer retains a borrowed config pointer.

Validation

• make -C c check - all C tests and 71 Python tests pass
• make -C c deepseek-v4 ARCH=x86-64-v3
• make -C c deepseek-v4 ARCH=native
• ASan + UBSan + LSan: test_safetensors_index, test_deepseek_v4, test_v4_ownership
• Full 48-shard model oracle with MEMORY_GB=32
  • teacher forcing: 19/19
  • greedy continuation: 8/8
  • continuation self-check: 8/8
  • DSpark on/off exact-length identity: OK
  • generated text: "The capital of France is Paris."
• Deterministic tiny DeepSeek V4 + one-stage DSpark fixture
• committed fixture: 1,253,414 bytes
• independent reference: Transformers 5.14.1 DeepseekV4ForCausalLM
• target teacher forcing: token-exact
• target greedy decode: exact IDs and exact length
• DSpark enabled/disabled: exact target identity
• 72-token prompt: crosses the internal 64-token prefill boundary
• repeated engine/session lifecycle: pass
• CI
• Linux x86-64: tiny target + DSpark token-exact oracle passed
• Windows UCRT64: tiny target + DSpark token-exact oracle passed
• macOS: make check passed; portable V4 infrastructure tests ran, while
  the x86-64-only V4 engine and tiny runtime oracle were explicitly gated off

The full-model fixture uses source=coli-self: it validates deterministic target-token reproduction and DSpark losslessness, not independent Hugging Face parity.

CUDA validation is not applicable because this PR does not add or modify a CUDA backend.

Scope

This PR intentionally does not include the proposed AVX2 kernel or dual-SSD mirror work; those should be reviewed separately.

Compatibility

• No production checkpoint, generated binary, or benchmark artifact is included
• The repository includes only a ~1.20 MiB deterministic tiny test fixture

[rajpratham1]

This is a very impressive contribution and clearly represents a significant amount of engineering work. The implementation covers a complete DeepSeek V4 CPU inference pipeline including runtime, expert streaming, quantization, safetensors loading, CLI tooling, documentation, and an extensive unit test suite.

Because this PR introduces an entirely new inference stack across many core components, I'd prefer additional review before approval.

Some areas that would benefit from closer review include:

• Long-term API stability for the new DeepSeek V4 interfaces.
• Memory ownership and lifetime throughout the expert streaming/runtime pipeline.
• Performance characteristics of the NVMe streaming implementation under sustained inference.
• Cross-platform compatibility (Windows/Linux/macOS) for filesystem and I/O paths.
• Validation against larger real-world models beyond the included unit tests.

Overall the direction looks very promising, but given the size and architectural impact of this change, I think it should receive another maintainer review before merging.

[DrewZt]

Thanks for the thoughtful review. I agree that another maintainer review is appropriate given the size of the change.

A few clarifications on the areas you mentioned:

API stability: the new interfaces are currently scoped to the DeepSeek V4 engine and should be considered experimental. They are not intended to establish a stable generic model API at this stage.
Memory ownership: the expert-store API uses explicit lookup/release semantics, and the unit tests cover cache reuse and resource accounting. I agree that this area deserves focused review, and I can add more ownership/lifetime documentation where the contracts are not clear enough.
Sustained NVMe performance: the current documentation contains single-run measurements. I still need to add repeated and longer-running tests, including cache behavior, disk throughput, and memory stability over sustained decoding.
Cross-platform support: the DeepSeek V4 engine is intentionally limited to x86-64 Linux and Windows/MSYS2 for now. macOS, PowerPC, and other platforms are gated out of the V4 build and continue to run the existing GLM checks unchanged.
Full-model validation: the engine has been exercised end to end with the actual DeepSeek-V4-Flash-DSpark checkpoint. The current oracle path validates deterministic target-token reproduction and DSpark on/off identity; comparison against the official Transformers implementation is supported by the tooling but still needs broader validation.

I’m happy to address targeted follow-up findings, add more validation, or split parts of the implementation into staged PRs if the maintainers feel that would make review and long-term maintenance easier.

[DrewZt]

I pushed a follow-up series through 82e7760 addressing the API-boundary and ownership concerns raised in the review.

The main changes are:

• the DeepSeek V4 engine/session API is now explicitly marked experimental;
• implementation-specific safetensors and ExpertStore interfaces have been moved out of the public API;
• model paths, indexes, expert stores, resident weights, head caches, and DSpark runtime state now have explicit engine ownership;
• engine/session lifetime accounting is shared by the production and ownership-test paths;
• partially initialized engines and DSpark runners are cleaned up correctly on failure;
• the ExpertStore lookup/release lease contract is documented and covered by regression tests;
• ownership fault-injection hooks and test objects are isolated from production builds;
• V4 session-owned tokenizer allocations are now released when the session is destroyed;
• the existing GLM tokenizer loading behavior and GLM runtime paths remain unchanged.

The latest changes pass:

make -C c check -j8

The remaining validation work is focused on sustained NVMe behavior and broader full-model comparison against the official Transformers implementation. Those limitations remain documented in the PR and do not represent unresolved API or resource-lifetime issues.

I’m marking the PR ready for review and would appreciate another maintainer look, particularly at the revised API boundary and ownership model.

[steve-m]

First full-model validation on Linux/x86-64 — plus a build fix and an AVX2 kernel series, branch ready to pull: steve-m/colibri@v4-avx2-kernels (5 commits on top of this PR's dev).

Hardware: Ryzen 9 3900X (Zen 2, 12C/24T, AVX2 no AVX-512), 62 GiB RAM, NVMe ~6 GB/s, Manjaro, gcc 15. Model: DeepSeek-V4-Flash-DSpark, --memory-gb 40.

Build fix you'll want regardless: two amalgam units (BLOCK_HYBRID, GENERATE_STATS) use pthreads without including pthread.h — gcc 14+ (C23) makes implicit declarations hard errors, so make deepseek-v4 fails out of the box on current Linux toolchains. One-line includes, first commit on the branch.

Kernel series (all behind a COLI_V4_AVX2=0 runtime kill-switch, float paths kept as fallbacks):

• FP4 experts: int8 dot via pshufb-LUT + maddubs (doubled E2M1 values are integers), exact int32 accumulation per 32-block. On random tensors this measures ~4× closer to an fp64 oracle than the shipped float path (rel L2 0.027 vs 0.106).
• BF16 head: 8-lane exact widening + zero-copy pass over the resident head (the elementwise coli_bf16_decode loop was 0.23 s/token on this box).
• FP8 attention: fp32 bits built directly from E4M3 fields with a branchless denormal blend; same QDQ-activation math and product grouping as the reference, differs only by summation order.

Measured, 48-token free-form decode, OMP_NUM_THREADS=12: 0.53 tok/s (PR defaults) → 0.62 (threads = physical cores; SMT only adds barrier traffic) → 0.78 (FP4 int8) → 0.99 (head) → 1.12 tok/s (FP8). Verify phase −27%, TTFT 20 → 16.7 s. Validation at every step: your oracle tool's continuation_self_check 8/8 and DSpark on/off identity OK (the batch kernels are bitwise-identical per item to the single matvec by construction, so speculation identity survives), plus a new tests/test_native_quant_avx2.c (fp64-oracle error bounds incl. denormal-only tensors, bitwise dual/batch==single, dispatch kill-switch).

Also on the branch, off by default: a dual-SSD mirror (COLI_V4_MODEL_MIRROR) that hash-routes reads across two model copies at the coli_st_read_at choke point. Honest data point: it's ~5% slower warm (routing splits the OS page cache across two copies) — it's there for cold starts and low-RAM boxes; docs in the commit message.

Happy to split any of this into separate PRs against your dev, or adjust to taste.

[DrewZt]

First full-model validation on Linux/x86-64 — plus a build fix and an AVX2 kernel series, branch ready to pull: steve-m/colibri@v4-avx2-kernels (5 commits on top of this PR's dev).

Hardware: Ryzen 9 3900X (Zen 2, 12C/24T, AVX2 no AVX-512), 62 GiB RAM, NVMe ~6 GB/s, Manjaro, gcc 15. Model: DeepSeek-V4-Flash-DSpark, --memory-gb 40.

Build fix you'll want regardless: two amalgam units (BLOCK_HYBRID, GENERATE_STATS) use pthreads without including pthread.h — gcc 14+ (C23) makes implicit declarations hard errors, so make deepseek-v4 fails out of the box on current Linux toolchains. One-line includes, first commit on the branch.

Kernel series (all behind a COLI_V4_AVX2=0 runtime kill-switch, float paths kept as fallbacks):

• FP4 experts: int8 dot via pshufb-LUT + maddubs (doubled E2M1 values are integers), exact int32 accumulation per 32-block. On random tensors this measures ~4× closer to an fp64 oracle than the shipped float path (rel L2 0.027 vs 0.106).
• BF16 head: 8-lane exact widening + zero-copy pass over the resident head (the elementwise coli_bf16_decode loop was 0.23 s/token on this box).
• FP8 attention: fp32 bits built directly from E4M3 fields with a branchless denormal blend; same QDQ-activation math and product grouping as the reference, differs only by summation order.

Measured, 48-token free-form decode, OMP_NUM_THREADS=12: 0.53 tok/s (PR defaults) → 0.62 (threads = physical cores; SMT only adds barrier traffic) → 0.78 (FP4 int8) → 0.99 (head) → 1.12 tok/s (FP8). Verify phase −27%, TTFT 20 → 16.7 s. Validation at every step: your oracle tool's continuation_self_check 8/8 and DSpark on/off identity OK (the batch kernels are bitwise-identical per item to the single matvec by construction, so speculation identity survives), plus a new tests/test_native_quant_avx2.c (fp64-oracle error bounds incl. denormal-only tensors, bitwise dual/batch==single, dispatch kill-switch).

Also on the branch, off by default: a dual-SSD mirror (COLI_V4_MODEL_MIRROR) that hash-routes reads across two model copies at the coli_st_read_at choke point. Honest data point: it's ~5% slower warm (routing splits the OS page cache across two copies) — it's there for cold starts and low-RAM boxes; docs in the commit message.

Happy to split any of this into separate PRs against your dev, or adjust to taste.

This is extremely helpful — thank you for doing the first independent full-model Linux/x86-64 validation and for documenting the performance progression in such detail. The Ryzen 3900X result directly addresses one of the main remaining validation gaps for this PR, and the 0.53 → 1.12 tok/s breakdown makes it much easier to see where the current bottlenecks are.

I checked the branch history and it looks like v4-avx2-kernels was based on f2ff5ad, which was the PR head when you started, rather than the current head 82e7760. The branches have since diverged, and the optimization branch is missing the API-boundary and resource-ownership follow-ups added after that snapshot.

Those later commits include the revised public/internal API separation, engine/session lifetime accounting, failure-path cleanup, isolated ownership-test objects, and V4 session tokenizer cleanup. The optimization work is still very valuable, but it should be rebased onto 82e7760 before integration so that those ownership changes are not accidentally overwritten or bypassed.

One clarification regarding the pthread build issue: I had already addressed the missing declaration problem in c9b626c, immediately after the snapshot your branch was based on, by adding -pthread -include pthread.h to the Linux V4 build flags. That fix is already present in the current PR head.

Your source-level includes may still be a cleaner and more localized solution, but the issue itself no longer needs a separate build-fix PR. When rebasing, please either drop the overlapping build-fix commit or call out why replacing the current compiler-level include with explicit includes in the two amalgam units would be preferable.

For the remaining work, I suggest splitting it into two focused follow-ups:

1. AVX2 kernel series

   Please keep the FP4 expert, BF16 head, and FP8 attention kernels together with tests/test_native_quant_avx2.c in a dedicated PR rebased onto 82e7760.

   Keeping the scalar/float fallbacks and the COLI_V4_AVX2=0 runtime kill switch is a good compatibility approach. The fp64 error-bound tests, denormal cases, batch-versus-single checks, and dispatch-disable coverage are especially useful.

2. Dual-SSD mirror

   I think COLI_V4_MODEL_MIRROR should remain a separate experimental PR. Its expected benefits and tradeoffs differ from the CPU kernels, and the warm-cache regression you measured is important context. It appears most relevant to cold starts and lower-memory systems rather than the default warm-cache configuration.

I also noticed that the branch diff appears to include generated test binaries such as:

c/tests/test_decode_batch
c/tests/test_i4_acc512
c/tests/test_idot

Please drop those artifacts when preparing the follow-up PRs.

After rebasing, please rerun the full-model correctness and performance validation because the runtime and ownership code has changed since f2ff5ad. The most useful checks would be:

make -C c check -j8
make -C c deepseek-v4

continuation_self_check
DSpark on/off identity
COLI_V4_AVX2=0 fallback comparison
48-token benchmark with the same model, RAM cap, and thread count

The current benchmark result is already valuable as independent validation of the earlier implementation. Once the rebased kernel series is reviewed, I’d be happy to add the reproducible Linux/x86-64 measurements to the documentation with credit to you and the exact hardware and commands.

Thanks again — this is a substantial and very useful contribution. Rebasing and splitting it should let us preserve the recent API and ownership work while giving the kernels and storage experiment the focused review they deserve.

[DrewZt]

Follow-up for @JustVugg: head af86de1 closes the remaining validation findings. The safetensors index now requires payload bytes to equal dtype width times shape numel; V4 config integers, floats, and compress ratios now have finite/integer/range checks. I also removed the dense-cache borrowed config pointer in favor of the engine-owned canonical config and made the DSpark oracle require exact output lengths on both paths. Fresh validation passed make check (all C tests plus 71 Python tests), x86-64-v3/native builds, ASan+UBSan+LSan, and the 48-shard MEMORY_GB=32 oracle (19/19 teacher forcing, 8/8 greedy, 8/8 continuation self-check, exact DSpark on/off identity). This supersedes my earlier blocker summary; AVX2 and dual-SSD work remain out of scope. Could you please review when convenient?

[steve-m]

Pushed the rebased AVX2 kernel series to steve-m/colibri:v4-avx2-kernels (force-updated onto the current PR head af86de1, the ownership/API series you pushed after 82e7760) — 3 commits on top. Rebased rather than replayed the old branch, so the public/internal API split, engine/session ownership, failure-path cleanup, and session tokenizer changes are all preserved underneath — make -C c check (incl. the test_v4_ownership suite) passes on top of the series.

Two changes from the old v4-avx2-kernels snapshot:

• Dropped the pthread build-fix commit. Your -pthread -include pthread.h in the Linux V4 CFLAGS already covers it, so the separate source-include commit is gone — the engine builds clean on gcc 15 without it. (I kept the compiler-flag approach; happy to add explicit #include <pthread.h> to the two amalgam units instead if you'd prefer the localized form, but it's not needed for the build.)
• Left the dual-SSD mirror out of this series. It's off-by-default and ~5% slower warm on a single-cache box, so it doesn't belong in the kernel PR; I'll keep it on a separate branch if there's interest.

The three commits, all behind the COLI_V4_AVX2=0 runtime kill-switch with the float paths kept as fallbacks:

1. FP4 experts — int8 dot (pshufb-LUT + maddubs; doubled E2M1 values are integers, exact int32 accumulation per 32-block). Drops in at the link level — native_quant_avx2.c provides coli_fp4_matvec_ref/coli_fp8_matvec_ref and wraps the renamed *_float_ref fallbacks, so no engine call sites change. The native_quant_* files were byte-identical to the snapshot, so this was a clean port.
2. FP8 attention — E4M3 (fp32 bits built directly from the E4M3 fields with a branchless denormal blend; same QDQ-activation math and product grouping as the reference, differs only by summation order).
3. BF16 head + zero-copy resident pass. A shared coli_v4_head_row_dot (8-lane exact bf16 widening + FMA across the batch) replaces the per-element coli_bf16_decode loops in both coli_v4_target_head_argmax_batch and the single-token head_argmax; when the head is resident in the engine head cache, it now dots the cached rows in place instead of re-reading per ROWS block.

Validation on the rebased branch — Ryzen 9 3900X (Zen 2, AVX2 no AVX-512), 62 GiB, gcc 16.1.1, DeepSeek-V4-Flash-DSpark, --memory-gb 40:

• make -C c check — all pass, including test_native_quant_avx2 (fp64-oracle error bounds incl. denormal-only tensors, bitwise dual/batch==single, dispatch kill-switch) and the full test_v4_ownership suite. Builds clean on gcc 16 (only pre-existing warnings in deepseek_v4_dspark.c, none from the kernels).
• make deepseek-v4-oracle (coli-self): continuation_self_check 8/8, teacher-forcing 19/19 positions, greedy 8/8 tokens, and DSpark on/off identity OK (no_dspark vs fixture, dspark vs fixture, on/off equality all OK). The head is resident (head=resident-bf16), so the zero-copy resident-head path is the one exercised.

Perf — AVX2 on vs the COLI_V4_AVX2=0 float fallback on this rebased head (48-token free-form, OMP_NUM_THREADS=12).

Kernel-only isolation, DSpark disabled (--no-dspark) — both runs issue an identical, deterministic 18318 expert requests, so this is a true A/B with no speculation-acceptance variance:

• decode 0.76 → 0.98 tok/s (+29%)
• prefill / TTFT 48.2 s → 31.2 s (−35%)

For reference, DSpark on (same prompt): decode 0.62 → 0.85 tok/s (+36%), TTFT 43.6 → 30.3 s (−30%) — but that pass had asymmetric acceptance (on 3/10 vs off 1/10 speculative tokens), so the isolated --no-dspark figure above is the honest kernel-only claim.

The float-path baseline matches the known physical-cores number, so the kill-switch cleanly isolates the kernel gain. TTFT is stable across the DSpark condition (~31 s on / ~48 s off), as expected — the first token is prefill + first decode, before any speculation. (Absolute tok/s is workload-dependent — this was a heavy ~80 GB-streaming pass; a lighter/warmer pass on this box peaks around 1.12 tok/s. The on/off ratio is the stable claim.)

[DrewZt]

Pushed the rebased AVX2 kernel series to steve-m/colibri:v4-avx2-kernels (force-updated onto the current PR head af86de1, the ownership/API series you pushed after 82e7760) — 3 commits on top. Rebased rather than replayed the old branch, so the public/internal API split, engine/session ownership, failure-path cleanup, and session tokenizer changes are all preserved underneath — make -C c check (incl. the test_v4_ownership suite) passes on top of the series.

Two changes from the old v4-avx2-kernels snapshot:

• Dropped the pthread build-fix commit. Your -pthread -include pthread.h in the Linux V4 CFLAGS already covers it, so the separate source-include commit is gone — the engine builds clean on gcc 15 without it. (I kept the compiler-flag approach; happy to add explicit #include <pthread.h> to the two amalgam units instead if you'd prefer the localized form, but it's not needed for the build.)
• Left the dual-SSD mirror out of this series. It's off-by-default and ~5% slower warm on a single-cache box, so it doesn't belong in the kernel PR; I'll keep it on a separate branch if there's interest.

The three commits, all behind the COLI_V4_AVX2=0 runtime kill-switch with the float paths kept as fallbacks:

1. FP4 experts — int8 dot (pshufb-LUT + maddubs; doubled E2M1 values are integers, exact int32 accumulation per 32-block). Drops in at the link level — native_quant_avx2.c provides coli_fp4_matvec_ref/coli_fp8_matvec_ref and wraps the renamed *_float_ref fallbacks, so no engine call sites change. The native_quant_* files were byte-identical to the snapshot, so this was a clean port.
2. FP8 attention — E4M3 (fp32 bits built directly from the E4M3 fields with a branchless denormal blend; same QDQ-activation math and product grouping as the reference, differs only by summation order).
3. BF16 head + zero-copy resident pass. A shared coli_v4_head_row_dot (8-lane exact bf16 widening + FMA across the batch) replaces the per-element coli_bf16_decode loops in both coli_v4_target_head_argmax_batch and the single-token head_argmax; when the head is resident in the engine head cache, it now dots the cached rows in place instead of re-reading per ROWS block.

Validation on the rebased branch — Ryzen 9 3900X (Zen 2, AVX2 no AVX-512), 62 GiB, gcc 16.1.1, DeepSeek-V4-Flash-DSpark, --memory-gb 40:

• make -C c check — all pass, including test_native_quant_avx2 (fp64-oracle error bounds incl. denormal-only tensors, bitwise dual/batch==single, dispatch kill-switch) and the full test_v4_ownership suite. Builds clean on gcc 16 (only pre-existing warnings in deepseek_v4_dspark.c, none from the kernels).
• make deepseek-v4-oracle (coli-self): continuation_self_check 8/8, teacher-forcing 19/19 positions, greedy 8/8 tokens, and DSpark on/off identity OK (no_dspark vs fixture, dspark vs fixture, on/off equality all OK). The head is resident (head=resident-bf16), so the zero-copy resident-head path is the one exercised.

Perf — AVX2 on vs the COLI_V4_AVX2=0 float fallback on this rebased head (48-token free-form, OMP_NUM_THREADS=12).

Kernel-only isolation, DSpark disabled (--no-dspark) — both runs issue an identical, deterministic 18318 expert requests, so this is a true A/B with no speculation-acceptance variance:

• decode 0.76 → 0.98 tok/s (+29%)
• prefill / TTFT 48.2 s → 31.2 s (−35%)

For reference, DSpark on (same prompt): decode 0.62 → 0.85 tok/s (+36%), TTFT 43.6 → 30.3 s (−30%) — but that pass had asymmetric acceptance (on 3/10 vs off 1/10 speculative tokens), so the isolated --no-dspark figure above is the honest kernel-only claim.

The float-path baseline matches the known physical-cores number, so the kill-switch cleanly isolates the kernel gain. TTFT is stable across the DSpark condition (~31 s on / ~48 s off), as expected — the first token is prefill + first decode, before any speculation. (Absolute tok/s is workload-dependent — this was a heavy ~80 GB-streaming pass; a lighter/warmer pass on this box peaks around 1.12 tok/s. The on/off ratio is the stable claim.)

This looks excellent — thank you for rebasing the series carefully and for preserving the API and ownership work underneath it.

The updated scope is exactly what I was hoping for:

• the overlapping pthread fix is removed;
• the dual-SSD experiment is kept separate;
• the AVX2 work is limited to three focused kernel commits;
• the float fallbacks and COLI_V4_AVX2=0 escape hatch remain available;
• the ownership suite and full-model oracle still pass on top of the series.

I also appreciate the distinction between the DSpark-enabled result and the --no-dspark isolation run. The deterministic 18,318-request A/B is the right primary performance claim because it removes speculative-acceptance variance. The resulting figures are both substantial and clearly scoped:

decode:       0.76 → 0.98 tok/s  (+29%)
prefill/TTFT: 48.2 → 31.2 s     (-35%)

The zero-copy resident-head path being exercised by the full-model oracle is also useful confirmation that the new path is covered rather than only compiled.

Since the kernel branch is stacked on top of #165, I think the cleanest next step is to keep the branch as-is for now and open a dedicated follow-up PR after #165 is merged. At that point it can be rebased onto the resulting upstream dev, leaving the follow-up diff limited to the three kernel commits and their tests.

Please preserve the current commit separation and validation details in that PR. In particular, the --no-dspark A/B should be the headline benchmark, with the DSpark-enabled figures included as additional workload-dependent context.

Thanks again — this is a strong follow-up series, and the careful validation and honest performance attribution make it much easier to review.

[DrewZt]

Pushed follow-up stability fix e4f87a8 for two issues found during full-model testing:

• Fixed prompts longer than the internal 64-token batch limit by chunking target-layer and DSpark prefill while preserving absolute positions.
• Fixed DSpark sparse-window accounting after speculative position jumps. The previous monotonically incremented valid counter could report more entries than were actually copyable (valid=43 copied=42), resulting in DSpark block failed.
• Added stage-specific DSpark diagnostics and a regression test for sparse absolute-position window updates.

Validation:

• The previous 74/76-token failure cases now complete successfully.
• A near-limit 488-token prompt completed prefill and generated its first token successfully.
• The DeepSeek V4 unit tests pass.
• make deepseek-v4 succeeds.

[JustVugg]

I'm interested in this — DeepSeek V4 Flash on CPU with NVMe expert streaming is squarely what colibrì is for, and I'd like it in.

The one condition is that I need to run it on my own machine first. Not as a gate to be difficult: it's the rule I've had to learn the hard way this week. EXPERT_BUDGET went in on numbers nobody had reproduced and had to be quarantined a few days later (#303) — it turned out to be slower than not using it while claiming a speedup. I'm not going to do that to a second engine. Once something is in main, people run it, and if I can't run it I can't fix it for them.

So: as soon as I can get a checkpoint on this box, I'll test it and we'll work on it together. That's not a "no" parked forever — it's the next thing I want to do on this front.

Two things that would make it land sooner, and I'd rather ask than have you guess:

1. A path to a small test model. The pattern already in the repo is tools/make_glm_oracle.py, which builds a tiny glm_tiny/ that the engine is scored token-exact against. If DeepSeek V4 can get the same — even a toy — then the engine proves itself on every CI run, on my box and everyone else's, without a 400 GB download. That's the single highest-leverage thing here: it turns "trust me" into "the test is green", and it's what will keep the engine alive in six months when neither of us is looking at it.
2. Keep simplifying the implementation. 8,704 lines is a lot to carry, and the thinner it gets the faster it moves — for both of us. Anything that can lean on what glm.c already has (the streaming cache, the URING/DIRECT I/O path, compat.h) rather than reimplement it is line I don't have to review and you don't have to maintain. c/compat.h +7 is exactly the right kind of touch.

I saw the stability fix you pushed (e4f87a8) — thank you for staying on it. And CI landed today (#143 + #144): make check now runs on ubuntu/windows/macos for every PR, so you'll get a compile/test verdict in ~2 minutes instead of waiting on me.

Keeping this open. Let's keep going.

[DrewZt]

I'm interested in this — DeepSeek V4 Flash on CPU with NVMe expert streaming is squarely what colibrì is for, and I'd like it in.

The one condition is that I need to run it on my own machine first. Not as a gate to be difficult: it's the rule I've had to learn the hard way this week. EXPERT_BUDGET went in on numbers nobody had reproduced and had to be quarantined a few days later (#303) — it turned out to be slower than not using it while claiming a speedup. I'm not going to do that to a second engine. Once something is in main, people run it, and if I can't run it I can't fix it for them.

So: as soon as I can get a checkpoint on this box, I'll test it and we'll work on it together. That's not a "no" parked forever — it's the next thing I want to do on this front.

Two things that would make it land sooner, and I'd rather ask than have you guess:

1. A path to a small test model. The pattern already in the repo is tools/make_glm_oracle.py, which builds a tiny glm_tiny/ that the engine is scored token-exact against. If DeepSeek V4 can get the same — even a toy — then the engine proves itself on every CI run, on my box and everyone else's, without a 400 GB download. That's the single highest-leverage thing here: it turns "trust me" into "the test is green", and it's what will keep the engine alive in six months when neither of us is looking at it.
2. Keep simplifying the implementation. 8,704 lines is a lot to carry, and the thinner it gets the faster it moves — for both of us. Anything that can lean on what glm.c already has (the streaming cache, the URING/DIRECT I/O path, compat.h) rather than reimplement it is line I don't have to review and you don't have to maintain. c/compat.h +7 is exactly the right kind of touch.

I saw the stability fix you pushed (e4f87a8) — thank you for staying on it. And CI landed today (#143 + #144): make check now runs on ubuntu/windows/macos for every PR, so you'll get a compile/test verdict in ~2 minutes instead of waiting on me.

Keeping this open. Let's keep going.

Thanks — this gives me a clear integration path.

I’ll make the tiny DeepSeek V4 fixture the immediate priority. The goal will be a deterministic, independently generated checkpoint and reference that make check can validate without downloading the full model or installing PyTorch/Transformers during CI.

The initial test contract will cover:

• target teacher forcing against the independent reference;
• target greedy decoding against the same reference;
• DSpark-off output matching the target reference exactly;
• DSpark-on output remaining exactly identical to the target path;
• compressed-attention and routed-expert execution;
• a prompt longer than the internal 64-token batch boundary, covering the recent chunked-prefill fix.

The generator will remain available so the fixture is reproducible rather than an opaque committed artifact, while CI itself will use the checked-in tiny checkpoint and reference.

I’ll also continue removing local duplication where that does not change runtime or kernel boundaries. For the deeper shared-I/O work—particularly consolidating the streaming read path with the existing DIRECT/URING infrastructure—I would prefer to do that as a focused follow-up after this base engine and the already prepared AVX2 series land. That keeps the current correctness and performance baselines stable, avoids repeatedly invalidating the AVX2 branch, and lets the tiny token-exact oracle protect the later refactor across both scalar and optimized paths.

I’ll keep AVX2 and the dual-SSD experiment out of this PR, and I’ll document the shortest full-checkpoint smoke-test path for your machine alongside the tiny fixture.

[DrewZt]

Implemented the requested deterministic tiny DeepSeek V4 + DSpark oracle at head dac2f76.

• Committed fixture size: 1,253,414 bytes (~1.20 MiB), including target and one-stage DSpark checkpoints.
• Independent reference: official Transformers DeepseekV4ForCausalLM (Transformers 5.14.1), evaluated after the same dense FP8, routed-expert packed FP4, and BF16 round trips loaded by the C runtime. The C engine is never used to generate the reference.
• Target teacher forcing: token-exact for the short, compressed-attention, and 72-token prompts.
• Target greedy decode: exact IDs and exact length; truncated-prefix comparisons are explicitly rejected.
• DSpark identity: drafting is exercised and enabled/disabled outputs exactly match the independent target reference.
• Long prompt: the 72-token case crosses the internal 64-token prefill boundary for both target and DSpark.
• Lifecycle: repeated engine/session open, generate, destroy checks pass.
• CI: Linux x86-64 and Windows UCRT64 run the tiny oracle in make check; macOS passes the normal GLM tests and V4 platform gate. All three jobs passed: https://github.com/whale-agent-lab/colibri/actions/runs/29556719562
• CI remains offline and does not install PyTorch or Transformers.

Manual local validation (from the repository root):

# Dedicated tiny target + DSpark token-exact oracle
make -C c deepseek-v4-tiny-check ARCH=x86-64-v3

# Or run the complete dependency-free check suite
make -C c check

To exercise the normal runtime explicitly with drafting disabled:

cd c
make deepseek-v4 ARCH=x86-64-v3
./deepseek_v4 deepseek_v4_tiny '<t005><t007><t009>' \
  --raw-prompt --draft-model deepseek_v4_tiny/dspark --no-dspark

The generator and regeneration package versions are documented in docs/deepseek-v4.md. AVX2/native-quant dispatch redesign and shared DIRECT/URING/read-helper consolidation remain out of scope.

[maikelthedev]

1.12 tok/s (FP8).

sorry what?! That's impressive for NVMe.

[DrewZt]

1.12 tok/s (FP8).

sorry what?! That's impressive for NVMe.

The prompt 'What is the capital of France' have some uniqueness, the answer token got 100% dspark acceptance rate, that is the fastest speed boost.
For another 0% dspark acceptance rate special case 'hello', decode speed will drop to 0.59tok/s, which is slower than no-dspark situation

[maikelthedev]

On your own Pc, Vincent, how does it compare to GLM in speed if you don't mind me asking (And what are the specs of yours , and what OS) 🤔From: DrewZt ***@***.***>Sent: Saturday, July 18, 2026 12:07:24 pmTo: JustVugg/colibri ***@***.***>Cc: Maikel Frias Mosquea ***@***.***>; Comment ***@***.***>Subject: Re: [JustVugg/colibri] Add DeepSeek V4 Flash CPU inference with NVMe expert streaming (PR #165)DrewZt left a comment (JustVugg/colibri#165)1.12 tok/s (FP8).sorry what?! That's impressive for NVMe.The prompt 'What is the capital of France' have some uniqueness, the answer token got 100% dspark acceptance rate, that is the fastest speed boost. For another 0% dspark acceptance rate special case 'hello', decode speed will drop to 0.59tok/s, which is slower than no-dspark situation—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.

[DrewZt]

On your own Pc, Vincent, how does it compare to GLM in speed if you don't mind me asking (And what are the specs of yours , and what OS) 🤔From: DrewZt @.>Sent: Saturday, July 18, 2026 12:07:24 pmTo: JustVugg/colibri @.>Cc: Maikel Frias Mosquea @.>; Comment @.>Subject: Re: [JustVugg/colibri] Add DeepSeek V4 Flash CPU inference with NVMe expert streaming (PR #165)DrewZt left a comment (JustVugg/colibri#165)1.12 tok/s (FP8).sorry what?! That's impressive for NVMe.The prompt 'What is the capital of France' have some uniqueness, the answer token got 100% dspark acceptance rate, that is the fastest speed boost. For another 0% dspark acceptance rate special case 'hello', decode speed will drop to 0.59tok/s, which is slower than no-dspark situation—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.

My pc is ai max 395+128gb ram+6gb/s ssd, tested glm with config think=0 and mtp=1, got around 0.7 tok/s

[JustVugg]

Status check: dev has moved substantially since this was opened (#391 refactor: glm.c → colibri.c + header modules, plus today's CUDA kernel rework in #298). A DeepSeek engine is very much on the roadmap, so this PR is interesting — but the rebase at this distance is a rewrite-sized job only the author can drive. Are you still working on it? If not I'll close it as superseded when we start the DeepSeek port, with credit for the groundwork.

[DrewZt]

Status check: dev has moved substantially since this was opened (#391 refactor: glm.c → colibri.c + header modules, plus today's CUDA kernel rework in #298). A DeepSeek engine is very much on the roadmap, so this PR is interesting — but the rebase at this distance is a rewrite-sized job only the author can drive. Are you still working on it? If not I'll close it as superseded when we start the DeepSeek port, with credit for the groundwork.

Yes, I’m still actively working on this. Please keep the PR open.

I reviewed the impact of the current dev changes, including the glm.c → colibri.c refactor and the newer build, I/O, tokenizer, and quantization infrastructure. A commit-by-commit rebase would be unnecessarily difficult, but the V4 engine itself does not need to be rewritten: most of the model-specific implementation remains isolated in files that do not exist upstream.

I’m going to treat this as a minimal forward port onto current dev, rather than replaying the old history or combining it with another large refactor.

Work I will complete before merging #165

1. Port the independent V4 engine onto current dev

   • re-add the target and DSpark engine files on top of the current tree;
   • adapt the build targets, tests, cleanup rules, and platform gates to the current colibri build structure;
   • preserve the existing experimental engine/session API and ownership boundaries.

2. Reconcile the shared-file changes manually

   • keep the current upstream compat.h as the base and add only V4-specific missing pieces;
   • combine the current tokenizer/o200k changes with the V4 tokenizer lifetime cleanup;
   • preserve the safetensors validation and JSON ownership fixes;
   • avoid replacing newer upstream files with their older Add DeepSeek V4 Flash CPU inference with NVMe expert streaming #165 versions.

3. Keep the V4-specific runtime boundaries stable

   • retain the current V4 safetensors index and tensor I/O layer for this port;
   • retain the native FP4/FP8 quantization interfaces;
   • retain the ExpertStore, head-cache, target, and DSpark interfaces used by the validated implementation.

   The current upstream st.h and quant.h are useful foundations, but they do not yet represent the same tensor metadata or numerical formats as the V4 runtime. Folding those layers together during the port would substantially enlarge the correctness and review surface.

4. Restore the automated correctness gate first

   • make the committed tiny Transformers oracle pass on Linux and Windows;
   • keep the macOS unsupported-runtime gate and portable infrastructure tests;
   • restore teacher-forcing, greedy, DSpark identity, lifecycle, compressed-attention, and >64-token prompt coverage.

5. Repeat full-checkpoint validation

   • target teacher forcing;
   • greedy continuation;
   • DSpark enabled/disabled exact identity;
   • long-prompt prefill;
   • sustained decode and memory stability;
   • a fresh scalar --no-dspark performance baseline.

6. Keep AVX2 and the deeper I/O refactor out of the base port

   • I will not change the native-quant dispatch, head-cache representation, or kernel-facing APIs during the forward port;
   • this keeps Steve’s three AVX2 commits reusable rather than forcing him to redesign the kernels.

I had already done additional local work around redundant expert reads and the streamed I/O path, but I intentionally did not push it onto #165. At that point the branch had a stable correctness baseline and Steve’s AVX2 series was stacked on top of it; changing the I/O and cache boundaries again would have invalidated both his branch and the existing measurements.

Work planned after #165 merges

1. Steve’s AVX2 follow-up

   • rebase the focused FP4, FP8, and BF16-head kernel series onto the merged dev;
   • keep the scalar fallback and COLI_V4_AVX2=0 kill switch;
   • rerun the tiny oracle, full-model oracle, and deterministic --no-dspark A/B benchmark.

2. Shared I/O consolidation

   • evaluate the V4 read path against the current st.h, DIRECT I/O, pipeline, and io_uring infrastructure;
   • extract only genuinely shared low-level mechanisms such as full-read handling, direct/buffered fd management, prefetch, and platform diagnostics;
   • preserve the V4-specific tensor metadata, native FP4 layout, compressed attention, and cache semantics where they do not match the GLM engine.

3. Further implementation simplification

   • reduce duplicated infrastructure under the protection of both the tiny oracle and the AVX2 fallback tests;
   • keep that work in a focused follow-up rather than mixing model correctness, kernel optimization, and I/O refactoring into one review.

So the immediate objective is to restore #165 on current dev with the same validated model behavior and the smallest possible integration diff. Once that base is merged, the AVX2 and shared-I/O work can proceed as separate, measurable follow-ups.

[DrewZt]

The forward port is now pushed.

PR #165 now points to 9824fab, rebuilt as a fresh forward port on the current upstream dev at 4aca059.

The main colibri.c, storage/cache paths, and shared st.h/quant.h infrastructure remain unchanged. Steve’s AVX2 work and the broader I/O consolidation remain separate follow-ups.

Local make check, the tiny token-exact oracle, full-checkpoint validation, long-prompt tests, and sanitizer runs all pass.

The new PR workflows are currently waiting for maintainer approval. Could you approve them so the checks can run?

[JustVugg]

CI is green — both workflows, all jobs. Sorry that took a day: GitHub holds workflow runs from contributors whose first PR here hasn't merged yet, and I hadn't spotted how many were queued behind that setting. Nothing to do with your code.

So the state now: 20,780 lines, forward-ported onto current dev, and our own gates pass on it for the first time. That plus the fact that it touches neither colibri.c nor the shared headers — which is what made the forward port tractable in the first place — puts this in a very different position than it was in a week ago.

I owe you a real read of the engine before merging, which is the part I can't rush. Two things that would help while I do that:

1. Is the engine config-driven across V4 sizes? Your docs target Flash + DSpark (43 layers, hidden 4096, 256 experts). DeepSeek-V4-Pro is the same architecture at a different scale (61 layers, hidden 7168, 384 experts) with, as far as I can measure from the checkpoints, an identical tensor layout and quantization scheme — FP4 E2M1 experts with UE8M0 group-32 scales, FP8 E4M3 dense with 128×128 block scales. If Pro is "a bigger config" to your loader rather than new code, that's worth knowing early; I have the disk to try it and nobody else here does.
2. Anything you'd want reviewed first. Twenty thousand lines is a lot of surface; if there are two or three files where a second pair of eyes is most valuable, name them and I'll start there.

Thanks for staying with this through a refactor that would have killed most PRs.

[DrewZt]

CI is green — both workflows, all jobs. Sorry that took a day: GitHub holds workflow runs from contributors whose first PR here hasn't merged yet, and I hadn't spotted how many were queued behind that setting. Nothing to do with your code.

So the state now: 20,780 lines, forward-ported onto current dev, and our own gates pass on it for the first time. That plus the fact that it touches neither colibri.c nor the shared headers — which is what made the forward port tractable in the first place — puts this in a very different position than it was in a week ago.

I owe you a real read of the engine before merging, which is the part I can't rush. Two things that would help while I do that:

1. Is the engine config-driven across V4 sizes? Your docs target Flash + DSpark (43 layers, hidden 4096, 256 experts). DeepSeek-V4-Pro is the same architecture at a different scale (61 layers, hidden 7168, 384 experts) with, as far as I can measure from the checkpoints, an identical tensor layout and quantization scheme — FP4 E2M1 experts with UE8M0 group-32 scales, FP8 E4M3 dense with 128×128 block scales. If Pro is "a bigger config" to your loader rather than new code, that's worth knowing early; I have the disk to try it and nobody else here does.
2. Anything you'd want reviewed first. Twenty thousand lines is a lot of surface; if there are two or three files where a second pair of eyes is most valuable, name them and I'll start there.

Thanks for staying with this through a refactor that would have killed most PRs.

Thanks — and yes, the target engine is intended to be config-driven rather than tied to the Flash dimensions.

Layer count, hidden size, attention ranks, expert count, MoE width, compression schedule, and tensor shapes are all derived from config.json, with support for up to 128 target layers. Based on the Pro layout and quantization scheme you described, I expect it to use the same target path as a larger configuration rather than require a separate engine.

One important current limitation: please use the DeepSeek-V4-Flash-DSpark checkpoint for testing, not the plain Flash checkpoint. I found a checkpoint read issue yesterday, and the current branch only loads the DSpark-packaged layout correctly. Target-only execution can still be tested with --no-dspark; the limitation is the checkpoint package being loaded. I have not fixed the plain-checkpoint path yet.

I have not tested a Pro checkpoint yet, so I would treat that as expected compatibility rather than confirmed support. The main things to verify are tensor names, compression ratios, tokenizer/prompt format, memory planning, and the Pro DSpark stage layout if a draft checkpoint is available.

The most useful places to review first are:

• c/deepseek_v4.c: layer planning, target runtime, memory placement, and ownership;
• c/deepseek_v4_dspark.c: draft execution, verification, rollback, and state transitions;
• c/safetensors_index.c and c/tensor_io.c: checkpoint validation and read boundaries.

One additional note: the implementation still contains a non-trivial number of reference, fallback, and experimental paths that emerged during development. Some are useful correctness baselines or low-memory fallbacks; others may now duplicate the preferred path. I avoided broad pruning during the forward port to keep the validated behavior stable. After Flash and Pro validation, I’d like to classify them explicitly: retain tested fallbacks, isolate research-only variants, and remove purely historical duplication.

[DrewZt]

Pushed two follow-up commits, now at b09ab5:

--no-dspark is now a true target-only mode and no longer requires or initializes a DSpark checkpoint.
Added initial DeepSeek V4-Pro support for the non-DSpark target model.
Fixed a residency-planning bug affecting V4-Flash as well: dense and DSpark residency are now decided independently, so --no-dspark can still keep the target dense weights resident when memory allows.

Initial V4-Pro target-only performance under the current test configuration:

16 GiB memory limit: 0.087 tok/s
64 GiB memory limit: 0.24 tok/s

V4-Pro DSpark support has not been implemented; Pro currently requires --no-dspark.

[anrasi]

Tested this PR on aarch64 — first datapoint outside x86-64, I believe: NVIDIA DGX Spark GB10 (Grace, 10× Cortex-X925 + 10× Cortex-A725, 121 GB unified LPDDR5x), Ubuntu 24.04, gcc 13.3.0.

The engine core is genuinely portable: with the platform gates bypassed, 41 of 43 amalgam TUs compile clean on aarch64 with zero warnings. The only x86 dependencies were the two unconditional #include <immintrin.h> (their intrinsics were already behind __AVX512F__) and the platform gates themselves.

I put up a small branch on top of this PR's current head (2 commits, 6 files, +197/−15): whale-agent-lab#1. It:

1. Enables the build — guards the two immintrin includes with __AVX512F__, accepts aarch64/arm64 in the Makefile.deepseek-v4 uname gate, and adds an AARCH64+Linux branch next to the X86_64 one in the parent COLI_V4_SUPPORTED gate (macOS and Windows-on-ARM stay out). Without AVX-512 the engine takes the portable reference paths, exactly as on pre-AVX-512 x86.
2. Adds NEON rows16 kernels — aarch64 implementations of coli_fp4_matvec_rows16_v10 / coli_fp4_dual_matvec_rows16_v10 on the same 16-row interleaved layout (the 16-entry E2M1 table is exactly 64 bytes, so a four-register TBL gathers whole decoded floats), plus a COLI_FP4_ROWS16_KERNEL feature macro so the four __AVX512F__ gates in deepseek_v4.c key on kernel availability. Arithmetic is the same unfused (activation·value)·scale multiply-multiply-add per column, one row per vector lane, columns ascending — rows come out bit-identical to the scalar reference and to the AVX-512 kernel. x86 preprocessing is unchanged: with __AVX512F__ the macro is always defined and the same branches are selected.

Validation on the GB10, on top of b09ab5d:

• make -C c check green with the NEON kernels active: 33 C test binaries (including test_deepseek_v4, test_v4_ownership, test_v4_config_dspark_compat, which the gate now enables here), 137 Python tests, tiny oracle 11/11 token-exact (teacher forcing + greedy, sessions, DSpark on/off identity).
• Tiny fixture regenerated on-device with transformers 5.14.1 / torch 2.12.1 (real DeepseekV4ForCausalLM): reference tokens identical to the committed fixture, and the oracle passes against the regenerated fixture too.
• Standalone harness compiling the repo kernel sources against the flat scalar reference: outputs bit-identical across five shapes up to 7168×2048, single and dual, both nibble paths, with -ffp-contract=off and with default -O3.
• One coverage note that also applies to x86: the tiny fixture creates the hot policy with pin_slots_per_layer=0 (4 experts, all resident, so maximum_pins = slots_per_layer − min(experts_per_layer, 6) = 0), so the committed oracle never executes the rows16 kernels on any platform — kernel-level coverage above comes from the standalone harness.

Kernel bench at the real V4-Flash expert shapes (hidden 4096 × moe_inter 2048, FP4), flat OMP scalar (native_quant_parallel.c) vs NEON rows16, median of 15 on an idle machine:

shape	kernel	1 thread	10 threads	20 threads
2048×4096 (gate/up)	matvec	2.054 → 1.016 ms (2.02×)	1.49×	1.30×
2048×4096 (gate/up)	dual	4.500 → 1.912 ms (2.35×)	1.46×	1.13×
4096×2048 (down)	matvec	2.043 → 1.001 ms (2.04×)	1.51×	1.82×
4096×2048 (down)	dual	4.592 → 1.903 ms (2.41×)	1.70×	1.29×

Pack cost is ~2.9 ms per 4 MiB matrix, once per pinned expert. These are kernel-level numbers only — no full-model tokens/s yet (checkpoint download pending on this box).

Happy to split the build enablement from the NEON kernels if you'd rather keep this PR x86-only — whichever is easier.

[DrewZt]

Tested this PR on aarch64 — first datapoint outside x86-64, I believe: NVIDIA DGX Spark GB10 (Grace, 10× Cortex-X925 + 10× Cortex-A725, 121 GB unified LPDDR5x), Ubuntu 24.04, gcc 13.3.0.

The engine core is genuinely portable: with the platform gates bypassed, 41 of 43 amalgam TUs compile clean on aarch64 with zero warnings. The only x86 dependencies were the two unconditional #include <immintrin.h> (their intrinsics were already behind __AVX512F__) and the platform gates themselves.

I put up a small branch on top of this PR's current head (2 commits, 6 files, +197/−15): whale-agent-lab#1. It:

1. Enables the build — guards the two immintrin includes with __AVX512F__, accepts aarch64/arm64 in the Makefile.deepseek-v4 uname gate, and adds an AARCH64+Linux branch next to the X86_64 one in the parent COLI_V4_SUPPORTED gate (macOS and Windows-on-ARM stay out). Without AVX-512 the engine takes the portable reference paths, exactly as on pre-AVX-512 x86.
2. Adds NEON rows16 kernels — aarch64 implementations of coli_fp4_matvec_rows16_v10 / coli_fp4_dual_matvec_rows16_v10 on the same 16-row interleaved layout (the 16-entry E2M1 table is exactly 64 bytes, so a four-register TBL gathers whole decoded floats), plus a COLI_FP4_ROWS16_KERNEL feature macro so the four __AVX512F__ gates in deepseek_v4.c key on kernel availability. Arithmetic is the same unfused (activation·value)·scale multiply-multiply-add per column, one row per vector lane, columns ascending — rows come out bit-identical to the scalar reference and to the AVX-512 kernel. x86 preprocessing is unchanged: with __AVX512F__ the macro is always defined and the same branches are selected.

Validation on the GB10, on top of b09ab5d:

• make -C c check green with the NEON kernels active: 33 C test binaries (including test_deepseek_v4, test_v4_ownership, test_v4_config_dspark_compat, which the gate now enables here), 137 Python tests, tiny oracle 11/11 token-exact (teacher forcing + greedy, sessions, DSpark on/off identity).
• Tiny fixture regenerated on-device with transformers 5.14.1 / torch 2.12.1 (real DeepseekV4ForCausalLM): reference tokens identical to the committed fixture, and the oracle passes against the regenerated fixture too.
• Standalone harness compiling the repo kernel sources against the flat scalar reference: outputs bit-identical across five shapes up to 7168×2048, single and dual, both nibble paths, with -ffp-contract=off and with default -O3.
• One coverage note that also applies to x86: the tiny fixture creates the hot policy with pin_slots_per_layer=0 (4 experts, all resident, so maximum_pins = slots_per_layer − min(experts_per_layer, 6) = 0), so the committed oracle never executes the rows16 kernels on any platform — kernel-level coverage above comes from the standalone harness.

Kernel bench at the real V4-Flash expert shapes (hidden 4096 × moe_inter 2048, FP4), flat OMP scalar (native_quant_parallel.c) vs NEON rows16, median of 15 on an idle machine:

shape kernel 1 thread 10 threads 20 threads
2048×4096 (gate/up) matvec 2.054 → 1.016 ms (2.02×) 1.49× 1.30×
2048×4096 (gate/up) dual 4.500 → 1.912 ms (2.35×) 1.46× 1.13×
4096×2048 (down) matvec 2.043 → 1.001 ms (2.04×) 1.51× 1.82×
4096×2048 (down) dual 4.592 → 1.903 ms (2.41×) 1.70× 1.29×
Pack cost is ~2.9 ms per 4 MiB matrix, once per pinned expert. These are kernel-level numbers only — no full-model tokens/s yet (checkpoint download pending on this box).

Happy to split the build enablement from the NEON kernels if you'd rather keep this PR x86-only — whichever is easier.

This is excellent work — thank you for the thorough aarch64 validation and the focused NEON implementation.

The changes look reasonably contained, so I don’t think it makes sense to ask you to restructure them again unless the maintainers prefer a separate follow-up. I’ll keep the V4 runtime and kernel interfaces stable from this point so the branch does not need another rebase due to changes from my side.

I’m happy to leave the integration decision to @JustVugg: these commits can remain part of #165 if the added review scope is acceptable, or be moved to a focused follow-up if keeping the base PR x86-only would make the merge easier.

[anrasi]

Thanks @DrewZt — and the interface freeze is much appreciated. Either route works for me: whale-agent-lab#1 is ready as-is, and if @JustVugg prefers a focused follow-up I'll open one against dev as soon as that's the call. Next datapoint from this box will be full-model tokens/s once the checkpoint lands.

[DrewZt]

DeepSeek-V4-Flash-0731 works

[JustVugg]

Ok today i merge it!Works with coli chat and coli web too?

[DrewZt]

Ok today i merge it!Works with coli chat and coli web too?

Not directly yet.

This PR currently provides its own v4 chat launcher. It supports multi-turn conversation by rebuilding and re-prefilling the full text history on every turn, but it does not yet reuse or persist KV state across requests.

The experimental V4 engine/session API was designed with integration in mind, but it has not been wired into coli chat, coli serve, or the OpenAI-compatible backend used by coli web. The web frontend should be reusable once that backend adapter exists, but I have not tested that path yet.

I’d treat the coli chat / coli web integration and KV prefix reuse or persistence as follow-up work after this PR merges.

[pragg100510-lgtm]

DeepSeek-V4-Flash-0731 works

Hey @DrewZt, really appreciate you sticking with this PR through the refactor, the forward-port, and all the platform testing — it's been a genuinely impressive amount of follow-through.

Quick follow-up on the 'DeepSeek-V4-Flash-0731 works' update: could you share the Hugging Face URL for the exact checkpoint you tested? And if you have benchmark numbers from that run (tok/s, TTFT, etc.), would you mind posting them along with your PC specs (CPU, RAM, NVMe speed, OS)? Would help a lot in comparing against the earlier Ryzen 3900X / GB10 numbers in this thread.

Thanks again for all the work here!

[DrewZt]

DeepSeek-V4-Flash-0731 works

Hey @DrewZt, really appreciate you sticking with this PR through the refactor, the forward-port, and all the platform testing — it's been a genuinely impressive amount of follow-through.

Quick follow-up on the 'DeepSeek-V4-Flash-0731 works' update: could you share the Hugging Face URL for the exact checkpoint you tested? And if you have benchmark numbers from that run (tok/s, TTFT, etc.), would you mind posting them along with your PC specs (CPU, RAM, NVMe speed, OS)? Would help a lot in comparing against the earlier Ryzen 3900X / GB10 numbers in this thread.

Thanks again for all the work here!

the Hugging Face URL is deepseek-ai/DeepSeek-V4-Flash-0731
The test hardware and environment are documented in docs/deepseek-v4.md.

Using the same configuration—108 GiB RAM limit, 32 generated tokens, and two runs per model—the results were:

V4-Flash-0731: 0.868 tok/s average
DSpark checkpoint with --no-dspark: 0.821 tok/s average

The 0731 checkpoint was about 5.7% faster in decode. Average TTFT was effectively identical: 26.00s vs 26.12s. Average full-process wall time was 65.67s vs 68.20s, about 3.7% shorter for 0731.

Both runs used 186 target cache slots and a 99.58 GiB target cache.

[JustVugg]

DeepSeek-V4-Flash-0731 works is the headline here, and it changes what this PR is — it is no longer a port in progress, it is a working engine for a model people can download today. Thank you, and to @steve-m for the AVX2 kernel series and @anrasi for the datapoints.

I said earlier "today I merge it". I am walking that back, and you deserve the reasoning rather than silence, because most of it is our fault rather than yours — the tree moved underneath you while this was open.

Two things I got wrong about your PR, corrected first

I had this filed in my head as "duplicates c/json.h". It does not. You added json_free() to ours, fifteen lines, which is a contribution and not a duplicate. Sorry for the mischaracterisation.

I also described it as "seven .c files instead of the one-engine-file shape". That is unfair as stated: native_quant.c implements coli_e8m0_decode / coli_e2m1_decode / coli_e4m3fn_decode — real decoders you needed and that did not exist in this tree when you wrote them.

What changed underneath you, and why it matters

v1.4.0 shipped three days ago with two formats that land squarely on your file:

• fmt=7 — e2m1 nibbles + UE8M0 scale per 32 (#705, Kimi K3's native MXFP4)
• fmt=8 — fp8-e4m3 with 128×128 block scales (#528/#529, plus the FORMATS.md registry)

And here is why that is directly relevant. We measured DeepSeek-V4-Flash's actual shard headers rather than trusting config.json, and the checkpoint is:

	format	size
routed experts	e2m1 nibbles + UE8M0 per 32 → exactly fmt=7	157.4 GB
dense (attn, shared, o-proj)	fp8-e4m3, 128×128 blocks, UE8M0 scales → fmt=8 shape	8.4 GB
embed / lm_head	BF16	1.1 GB

The experts are declared I8 with shapes that read as literal but are byte arrays of nibble pairs — the nibble histogram settles it: codes k and k+8 match to within 0.5%, which is e2m1 sign symmetry and which int4 cannot produce. config.json does say expert_dtype: fp4 and scale_fmt: ue8m0, in fields that are easy to miss under the quantization_config block that describes only the dense half.

So most of native_quant.c now has an in-tree equivalent, and fmt=8 needs its UE8M0 scale variant implemented (FORMATS.md already registers it as "RECOGNIZED and refused by name, not yet implemented") — which is a much smaller job than carrying your own decode path, and it benefits every engine rather than this one.

What we would like, and it is mostly deletion

The shape we have settled on across inkling.c, kimi_k3.c and olmoe.c — and it is genuinely load-bearing now that CI builds all four engines on three platforms:

1. One engine file. c/deepseek_v4.c, the way kimi_k3.c is one file. It will still be large — your 8,792 lines are not padding, the hybrid attention and the hyper-connections are real work — but deepseek_v4_dspark.c, the six native_quant*.c, tensor_io.c and safetensors_index.c should fold in or go away.
2. st.h instead of safetensors_index.c + tensor_io.c. Shard indexing, pread streaming, the mirror/O_DIRECT paths and the OOB hardening from security: reject malformed model tensors at the untrusted-mirror boundary (C half of #368) #413 all live there already, and an engine that reads weights its own way misses every fix the others get.
3. quant.h for the decoders now that fmt=7 and fmt=8 exist.
4. No .safetensors in git. c/deepseek_v4_tiny/model.safetensors and the dspark/ one should be generated by a tools/make_tiny_deepseek.py, the way tools/make_tiny_inkling.py does — binaries in history never come back out.
5. Drop c/v4. The launcher is coli; a second entry point splits the surface that coli chat / serve / web all share.

And please wire the front end from the start

This is the part we would ask for even if nothing else changed. Users do not run engines directly — they run coli chat, coli web, coli serve. Two small edits get you all three:

# c/coli:160  model_arch()
if "deepseek" in model_type: return "deepseek"     # config.json says model_type: deepseek_v4

# c/coli:176  engine_for()
name = "inkling" if arch == "inkling" else "kimi_k3"   # -> add "deepseek_v4"

coli reads the model's config.json, picks the binary, and for non-GLM engines spawns the local gateway and attaches the TUI, so chat/web/serve/API all come from those two lines plus the chat template. Kimi K3 went through exactly this in #676 if you want a worked example.

DSpark

deepseek_v4_dspark.c is 4,057 lines of speculative decoding. Please leave it out of this PR and send it as a follow-up. Not because it is unwelcome — because a model that runs is the thing people are waiting for, and 4,000 lines of speculation review is what would keep this from landing for another week. Speculation also has to prove it earns its keep here (MTP is disabled on hosts where acceptance does not repay verification), so it wants its own measurements anyway.

What we can do to help

• We have the model downloading now and a 25 GB box to test the low-RAM path on, so we can give you real numbers rather than opinions.
• If the UE8M0 scale variant of fmt=8 is the annoying part, say so and we will implement it in quant.h as a separate PR so you can build on it.
• The .coli_usage routing telemetry is now shared (route_trace.h, The learning cache is engine-specific: .coli_usage has two incompatible writers and two engines that cannot produce it at all #700) — four call sites, and Kimi K3's adoption in feat(kimi_k3): adopt the shared routing telemetry (#700) — stacked on #716 #719 is the pattern.

No deadline from us, and nothing here is a complaint about the engineering. It is a request to land the same work in the shape the tree grew into while this PR was open. If some of it does not fit the way DeepSeek V4 actually works, say so — you have run this model and we have not.

[JustVugg]

if you want help i'm here too make this merge faster!

[JustVugg]

Correcting myself on DSpark — I asked you to hold it back, and the reasoning I gave was worse than the counter-argument.

What I said was that 4,057 lines of speculative decoding would keep a working engine from landing for another week. That is true of the review cost, and it is the wrong thing to optimise for.

DSpark is DeepSeek's, not an optimisation we invented. If it is part of how the model is meant to run, shipping without it hands people a half DeepSeek and asks them to judge it — and the first thing everyone will want to try is exactly the thing that is missing. Judging a model without the decoding path its authors designed for it is not a fair test of either.

So: please keep it, but as the second of two consecutive PRs rather than folded into one.

1. c/deepseek_v4.c — the engine, in the shape described above, with the coli wiring so chat/web/serve work from day one.
2. deepseek_v4_dspark — straight after, on top of the first.

Nothing gets thrown away, you do not rewrite anything, and neither of us reviews 13,000 lines in one sitting. If the first lands and the second is a day behind it, that is fine — people can run the model while the speculation is reviewed, instead of waiting for both.

One thing I would still ask of the DSpark PR, and it is the same standard MTP is held to here rather than anything specific to your work: numbers showing it earns its keep. Acceptance rate, tokens per forward, and the wall-clock delta with it on and off, on a host where you have run both. We disable native MTP on machines where acceptance does not repay verification, and DSpark should be measurable the same way — including a way to turn it off. Our own speculative work has been humbling: EXPERT_BUDGET is quarantined in colibri.c because it did more I/O than it saved, and MTP has measured a 32% loss at some cache-hit rates.

Take the time you need on both. Nothing here is urgent in a way that should push you into rushing a 9,000-line engine — and I would rather wait than have you strip something the model needs because a maintainer told you to.

[DrewZt]

Understood — I’ll keep the existing DSpark implementation, but split the work into two consecutive PRs as suggested.

For #165, I’ll focus on the target-only DeepSeek V4 engine, migrate it to the current shared st.h and quant.h infrastructure where applicable, remove the separate launcher and committed checkpoint binaries, and wire it into coli chat, coli serve, and coli web.

I’ll preserve the current full implementation on a separate branch, then prepare the DSpark support as an immediately stacked follow-up on top of the target-engine PR. That follow-up will retain --no-dspark and include acceptance rate, tokens-per-forward, and wall-clock on/off measurements under the same hardware and memory configuration.

The target engine was developed and validated independently before DSpark was added, so this split should preserve the existing work without requiring the target runtime to be redesigned.

[DrewZt]

For the target-engine cleanup, the main shared-infrastructure blocker is that upstream fmt=8 does not yet support the UE8M0 block scales used by DeepSeek V4. Could you add the UE8M0 scale variant to the shared quant.h implementation?

The required weight format is:

raw E4M3FN weight bytes in row-major [O, I] layout;
128×128 weight blocks;
one UE8M0 byte per block;
scale shape ceil(O / 128) × ceil(I / 128);
UE8M0 decoding: 0xff is NaN, otherwise 2^(value - 127).

It should coexist with the existing f32-scale fmt=8 path. Once that shared support is available, I’ll validate it against the current V4 implementation and the full DeepSeek-V4-Flash-0731 checkpoint, then remove the private FP8 weight path from #165.

I’ll keep the activation QDQ and the CPU routed-expert kernels V4-specific for now, since they are separate from the missing shared weight-format support.