CUDA Docker images v0.4.0 cannot initialize KVarN KV cache (kvarn4)

[vladimirdesyat]

Hi!

I believe this may be a packaging/build issue rather than a model issue.

The published CUDA Docker images for v0.4.0 fail to initialize KVarN KV cache. The same model loads successfully with regular KV cache types (for example q5_1), but fails when using kvarn4 for both K and V caches.

Environment:

• beellama.cpp v0.4.0
• Docker image: server-cuda13-v0.4.0 (also reproduced with server-cuda12-v0.4.0)
• GPU: RTX 4060 Ti 16GB
• GPU architecture: Ada Lovelace (compute capability 8.9)
• Model:
  • Qwen3.6-35B-A3B-MTP-GGUF
  • Source: Qwen3.6-35B-A3B-MTP-GGUF
  • File: Qwen3.6-35B-A3B-IQ4_XS-4.19bpw.gguf

I'm using router mode with ini configuration:

flash-attn = on
cache-type-k = kvarn4
cache-type-v = kvarn4

The server advertises KVarN cache types:

./llama-server --help | grep kvarn

Output includes:

kvarn2, kvarn3, kvarn4, kvarn5, kvarn6, kvarn8

However, when starting the server with this configuration, initialization fails:

load_model: loading model '/models/Qwen3.6-35B-A3B-IQ4_XS-4.19bpw.gguf'
llama_init_from_model: cannot enable kvarn_k4v4_g128: KVarN requires a backend with native KVarN FlashAttention support
common_fit_params: encountered an error while trying to fit params to free device memory: failed to create llama_context from model

The same error occurs with both CUDA 12 and CUDA 13 images.

I'm not sure whether this is related, but I noticed that the published binary reports:

./llama-server --version
version: 0 (unknown)
built with GNU 11.4.0 for Linux x86_64 (for cuda 13 'built with GNU 13.3.0 for Linux x86_64')

Is this a packaging/CI issue, or is an additional build option currently required for CUDA KVarN support?

[Anbeeld]

Please try v0.4.1 Preview: https://github.com/Anbeeld/beellama.cpp/releases/tag/preview-v0.4.1

It's supposed to fix these issues.

[vladimirdesyat]

Fixed in preview-v0.4.1.

I tested preview-v0.4.1 and the KVarN4 initialization issue seems to be resolved.

Previously, the model crashed during initialization when using KVarN cache types, but now it loads successfully with these settings:

cache-type-k = kvarn4
cache-type-v = kvarn3
cache-type-k-swa = kvarn4
cache-type-v-swa = kvarn3
kv-tail-tokens = auto
kv-tail-type = f16

Tested with short prompts so far.

I haven't yet tested long-context workloads (50k+ tokens / agent workflows) to verify stability.