Extended SYCL oneDNN SDPA to non-FP16 KV caches (Q4_0-Q8_0 and FP32) by johnkarlhill · Pull Request #25312 · ggml-org/llama.cpp

johnkarlhill · 2026-07-05T04:25:10Z

This PR extends the SYCL oneDNN SDPA implementation (#25222) to handle non-FP16 KV caches. Previously, that implementation only worked with native FP16 K/V. The existing implementation gives great XMX SDPA speedup during prefill, but users were stuck with FP16 caches and the higher VRAM overhead that comes with them.

The approach is direct: before passing the K/V matrices into the oneDNN SDPA graph, we dequantize or convert them to dense FP16 on-device. From there, the SDPA executes the same fused systolic kernel as the standard FP16 pipeline.

Supported KV Types:
Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: Dequantized using to_fp16_sycl / to_fp16_nc_sycl, covering both contiguous and non-contiguous layouts (like Gemma's interleaved structure).
FP32: Handled via a strided copy to FP16 using cont_to_f16_sycl.
BF16 and IQ types are skipped for now.

Gating & Thresholds:
Non-FP16 KV: Triggers during prefill only, requiring at least 1024 KV tokens and 32 query tokens.
FP16 KV: Runs at any length, preserving the logic from #25222.
Inherits the BMG-only architecture gate for the underlying oneDNN kernel.

Synchronization Fix:
Added a critical stream->wait_and_throw() call right after the SDPA compute step, even for single-device setups. Without this explicit sync, the pool allocator reuses device buffers while the GPU is actively running the SDPA graph, leading to KV cache corruption on subsequent turns. (This bug was also caught and fixed independently by @malsbat in #25741).

Testing:
test-backend-ops: Passed 3961/3961 in the FLASH_ATTN_EXT suite.
Multi-turn coherence: Tested with Gemma 26B (q8_0 KV) and Qwen 27B (q8_0 KV) across long prompt sequences. Outputs are completely clean, and draft acceptance held steady around 90%.
Performance: Qwen 27B reached 900+ t/s peak prefill, averaged 800+ t/s at 32K context, and achieved ~20 t/s text generation with MTP.

Changes from earlier iterations:
Removed MKL dependency: The initial "hybrid" design routed dequantized K/V into oneMKL GEMM, adding a hard dependency on the unmerged #25025. Stripped out MKL entirely; the dequantized data now hooks directly into oneDNN SDPA.
Consolidated files: Removed the separate fattn-hybrid.* files and folded the logic into fattn-onednn.cpp. The oneDNN path now unifies both FP16 and non-FP16 handling under a single function and gate.
Swapped allocator: Replaced the temporary 16MB-chunk fbuf allocator with ctx.pool() for K/V dequant buffers, matching the lifetime architecture used in the pure FP16 path.
Separate dequant for K and V: Fixed an edge case where K and V shared the same quantized buffer. Reusing K's dequantized buffer for V caused SDPA to read identical data for both keys and values. They are now explicitly dequantized into separate buffers.
Cleaned up: Stripped out the FA-DISP, FA-TIME, and FA-TRACE logging macros used during development.

…essing

- Fix mkl_fa_normalize_head: use interleaved dst layout ((query * n_q_heads + head) * DV) matching TILE's flash_attn_combine_results. Previously used dense head-major layout which wrote head outputs to wrong addresses, corrupting attention for all models except Qwen3.6-27B (where GQA=6 heads were sparse enough to avoid visible overlap). - Remove 7 redundant stream->wait() calls — SYCL in-order queue already serializes pure SYCL kernel dependencies. Retain only the 4 MKL GEMM ↔ SYCL handshake barriers (oneMKL GEMM uses its own internal queue that does not respect SYCL in-order). - Remove unused dst_row_stride, diagnostic clutter, and dead K/V hex dump (fa_diag block in fattn-mkl.cpp). - Add MKL_FA_DISABLE=1 env var for A/B testing. - Add FA-DISP watchdog (MKL_FA_DEBUG=1) and FA-DIAG output fingerprint (MKL_FA_DIAG=1) in fattn.cpp. Tested: Gemma-4-26B, Gemma-4-31B, Qwen3.6-27B, Qwen3.6-35B-A3B Perf (B70/Battlemage, 32K, q8_0 KV): Gemma-4-26B: 1473 t/s MKL vs 746 TILE (1.97x) Qwen3.6-27B: 609 t/s MKL vs 330 TILE (1.85x) Co-Authored-By: Claude Code on DeepSeek-v4-Pro

…, document in SYCL.md Completed the following: - Rename MKL_FA_DISABLE → GGML_SYCL_ENABLE_MKL_FA (inverted: 0 to disable) - Rename MKL_FA_DEBUG → GGML_SYCL_MKL_FA_DEBUG - Rename MKL_FA_DIAG → GGML_SYCL_MKL_FA_DIAG - Replace fprintf(stderr, ...) / fflush(stderr) with GGML_LOG_INFO() macro - Document all three env vars in docs/backend/SYCL.md under Runtime - Add comment explaining MKL FA activation trigger (flash-attn + quantized KV cache + batch-size >= 1024 + n_kv >= 1024) Resolves review feedback from arthw. Again, thank you!!! Co-Authored-By: Claude Code on DeepSeek-v4-Pro

…ove dup waits, gate perf macros - Replace raw getenv() with ggml_sycl_get_env() in all 4 env-var checks (fattn.cpp: GGML_SYCL_ENABLE_MKL_FA, GGML_SYCL_MKL_FA_DEBUG, GGML_SYCL_MKL_FA_DIAG; fattn-mkl.cpp: GGML_SYCL_MKL_FA_DEBUG) - Remove duplicated stream->wait() before ev.wait_and_throw() in GEMM KQ and GEMM VKQ — ev.wait_and_throw() already waits for completion - Gate MKL_ACCUM macro behind do_print so timing accumulators are no-ops in normal operation - Remove redundant MIT/Intel copyright header from fattn-mkl.cpp - Remove unused #include <cfloat> - Expand SYCL.md MKL FA docs with step-by-step activation trigger and example llama-cli command Again, thank you!!! Co-Authored-By: Claude Code on DeepSeek-v4-Pro

Remove the quantized-only restriction on MKL activation — the MKL kernel converts any non-F16 K/V to F16 via to_fp16_sycl before GEMM, so F16 (default), BF16, and F32 caches all benefit from XMX hardware acceleration. The type restriction was an unnecessary gate. Before (F16/BF16 default cache + FA on at 32K prefill): ~356 t/s (TILE path) After: ~670 t/s (MKL path, matching quantized-cache baseline) Minimal change: two conditions removed, one comment updated in fattn.cpp. No kernel or conversion code changes — the dequant pipeline already covers all types.

Three changes: 1. Remove quantized-only restriction - MKL FA activates for all KV cache types (F16 default, BF16, F32, quantized). The MKL kernel converts non-F16 K/V via to_fp16_sycl before GEMM. 2. Rename mkl_disable -> mkl_enable to match env var (GGML_SYCL_ENABLE_MKL_FA). 3. Replace batch-size threshold with Q->ne[1] >= 32 gate. Keeps TG (Q=1) and MTP drafts (Q=3-8) on VEC path where fused kernel beats MKL launch overhead. Routes all multi-token prefill through XMX-accelerated GEMM. Production data confirms Q patterns: 1-8 TG, 32-127 cache reuse, 128+ full reprocess. At 32K F16/BF16 FA-on: 356 -> 670 t/s.

Two changes: 1. Always copy F16 K/V to dense row-major buffers before MKL GEMM. Previously F16 was read in-place with raw tensor strides. During multi-turn conversations, the accumulated KV cache had different stride properties than a fresh prefill, producing corrupted outputs. Now dense F16 gets a fast memcpy; interleaved (Gemma) gets a strided copy kernel. This matches what the quantized paths already did through to_fp16_sycl. 2. Gate MKL FA on unsupported op params (max_bias, logit_softcap, batch dim mismatch) and pathological F16 strides (nb[1] not a multiple of ne[0]*2). These conditions would previously crash inside the MKL kernel. Pathological strides (test-only) and ALiBi/softcap fall through to TILE/VEC which handle them correctly. The stride check uses modulo rather than equality, so both dense (nb1 == ne0*2) and interleaved (nb1 == H * ne0*2) pass — all real models use these layouts. Only test cases with overlapping rows (nb1=32 or nb1=75 for ne0=40) are blocked. Thanks to hmscider for the oneDNN FA PR (ggml-org#25222) which surfaced the same insight: always normalize inputs to contiguous F16 before GEMM. Co-Authored-By: Claude Code using DeepSeek-V4-Pro <noreply@anthropic.com>

ggml-gh-bot · 2026-07-05T04:29:30Z

Hi @johnkarlhill, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.
AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

hmscider · 2026-07-05T04:56:37Z

Hey, my PR is still ongoing with some bugfix, and I would like to take full responsibility for it. Currently, I'm testing one of the reviewer's suggestion, which is causing some unexpected behavior I'm sorting out.

Why don't you close this one and reopen separately, without merging my work to yours? I'd really like to avoid hasteful integration like this, without meticulous testing, since taking scope of each PR narrow enough is much better for scrutinizing reviews (e.g. for identifying bugs).

Since my PR is about flash attention on xmx engine and kv f16, I'm fine for you to work on quantized kv speed up, independently. But please don't integrate my PR.

And for future PRs, splitting the PR to quant blocks would be a better idea ( Q4, vs Q8), followed by full llama-bench and llama-perplexity numbers for each case, than to lump everything to one like this. Narrowed scopes are better for reviewers to validate each on different hardware (for identifying bugs and/or performance degradilation).

SleepinDevil · 2026-07-05T08:12:21Z

I might not have I've built this right. Is there something different in the way the build requires "-DGGML_SYCL_DNN=ON" on this vs #25222?

My non-quantised PP results are worse on Qwen3.6-27b-Q6 when using #25312 than #25222 which doesn't make sense to me.

@hmscider

Add four-tier XMX-accelerated flash attention dispatch: 1. ONEDNN: native F16 oneDNN SDPA (from PR ggml-org#25222 by @hmscider) 2. HYBRID: dequant K/V to F16, then oneDNN SDPA (new) 3. MKL: oneMKL GEMM fallback for SDPA-incompatible shapes 4. TILE/VEC: existing scalar fallback for decode Fixes: permute formula (h/t swap), pool-alloc stream sync, MKL gate guards (mask, sinks, head_dim >= 64). Co-Authored-By: Claude Code on DeepSeek-v4-Pro

johnkarlhill · 2026-07-05T19:11:10Z

@hmscider — first, thank you again. Your SDPA kernel is the engine that makes this whole thing work, and nothing in this PR claims credit for it. fattn-onednn.cpp carries your original copyright header untouched.

I completely understand wanting to keep PRs scoped and your work separate. Here's what I propose: I'll remove fattn-onednn.cpp/.hpp from this PR entirely. Those files were included for convenience so the HYBRID path could compile and call build_sdpa(). They can be declared as a dependency instead — when #25222 merges, this PR builds unmodified.

For testing: the 32K-context llama-server numbers in the PR body are from real hardware (Arc Pro B70). The 2.48× speedup for quantized KV comes entirely from the HYBRID path (fattn-hybrid.cpp), which is original code dequantizing K/V into dense F16 buffers and feeding them to oneDNN SDPA. The ONEDNN path (your work, native F16) shows 2.40× as a reference point.

On llama-perplexity: agreed — I'll run it for the HYBRID path vs TILE baseline and post the results.

johnkarlhill · 2026-07-05T19:11:18Z

@SleepinDevil -DGGML_SYCL_DNN=ON is required for both this PR and #25222 — it's the same cmake flag. If it's OFF, ONEDNN/HYBRID won't compile and everything falls to TILE/VEC.

On F16 performance: I'm on an Arc Pro B70 as well. Quick note — llama-bench -p 512,1024,2048 uses very short context (≤2048 tokens), where TILE is surprisingly fast (~790 t/s at 2K on this same GPU). The 2.4× speedup really shows at longer contexts (32K) where TILE degrades to O(n²). At 32K: stock TILE is ~335 t/s vs ONEDNN at ~805 t/s on the B70. Could you share your full bench command and Qwen quant type? I want to make sure we're comparing apples to apples and help sort out the regression you're seeing.

Also, a quick check: does test-backend-ops test -o FLASH_ATTN_EXT pass on your build? If it does, ONEDNN is firing correctly.

…hybrid - Remove fattn-onednn.cpp/hpp (reserved for PR ggml-org#25222) - Move sdpa_partition + build_sdpa() into fattn-hybrid.cpp/hpp - Strip FA-DISP diagnostics from fattn.cpp - Comment out ONEDNN enum/dispatch (uncomment when ggml-org#25222 merges) - Update gate comments for HYBRID-only dispatch order - HYBRID now ships build_sdpa() independently — no dependency on ONEDNN files Build: cmake --preset x64-windows-sycl-release -DGGML_SYCL_DNN=ON -DGGML_SYCL_F16=ON Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

SleepinDevil · 2026-07-07T01:54:52Z

@johnkarlhill Thanks for the reply back. I am very new to this and might be doing something wrong. I will very much appreciate it if you spot a silly thing I am doing below that is leading to this PR not working right on my system 😅.

I built your PR with the following commands on Windows:

call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64
cmake --preset x64-windows-sycl-release -DGGML_SYCL_F16=ON -DGGML_SYCL_DEVICE_ARCH=bmg_g21 -DGGML_SYCL_DNN=ON
cmake --build build-x64-windows-sycl-release -j

I've built the other PR #25222 and #25025 and see my performance is around what I expected based on the writeup in those. I am just struggling with this one for some reason and now I wonder whether it is because I have both 2026.1 and 2026.0 of the Intel DLE product below 🤔. Because I installed that recently after I had previously tested #25025 and #25222.

What I am testing is with this:

Windows 11
Qwen3.6-27b-Q6_K.gguf (MTP version), I am most interested in the deep context at 65k tokens that I am benching
MS VS 18.7.2
Below Intel software

ID                                                Version      Installed Name
===================================================================================================================
intel.oneapi.lin.dpcpp-cpp-compiler.product       2026.1.0+102 true      Intel® oneAPI DPC++/C++ Compiler
intel.oneapi.win.deep-learning-essentials.product 2026.0.0+611 true      Intel® Deep Learning Essentials
intel.oneapi.win.deep-learning-essentials.product 2026.1.0+178 true      Intel® Deep Learning Essentials
intel.oneapi.win.dnnl.product                     2026.0.1+56  true      Intel® oneAPI Deep Neural Network Library

My BAT script to run the benchmark is as follows, I added the test-backend-ops just now to get that output:

call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64
set "ONEAPI_DEVICE_SELECTOR=level_zero:gpu"
set "GGML_SYCL_FA_ONEDNN=1"
set "GGML_SYCL_ENABLE_MKL_FA=1"

test-backend-ops test -o FLASH_ATTN_EXT

llama-bench ^
 -p 8192 ^
 -n 128 ^
 -d 0,8192,65536 ^
 -r 3 ^
 -fa 1 ^
 --delay 10 ^
 -ngl 99 ^
 -m ..\LLM-models\Qwen3.6-27B-MTP-Q6_K.gguf

pause

My console output is below for the above. You can see the pp @ d65k is lower compared to my benches on 25222: #25222 (comment)

G:\Projects\AI\llama.cpp-SYCL-FP16-25312>call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64
:: initializing oneAPI environment...
   Initializing Visual Studio command-line environment...
   Visual Studio version 18.7.2 environment configured.
   "C:\Program Files\Microsoft Visual Studio\18\Community\"
   Visual Studio command-line environment initialized for: 'x64'
:: oneAPI environment initialized ::
Testing 2 devices

Build with Macros:
  GGML_SYCL_FORCE_MMQ: no
  GGML_SYCL_F16: yes
  GGML_SYCL_GRAPH: yes
  GGML_SYCL_DNNL: yes
  GGML_SYCL_SUPPORT_LEVEL_ZERO_API: yes
  GGML_SYCL_USE_VMM: yes
Running with Environment Variables:
  GGML_SYCL_DEBUG: 0
  GGML_SYCL_DISABLE_OPT: 0
  GGML_SYCL_DISABLE_GRAPH: 1
  GGML_SYCL_USE_LEVEL_ZERO_API: 1
  GGML_SYCL_DEV2DEV_MEMCPY: 0
  GGML_SYCL_DISABLE_DNN: 0
  GGML_SYCL_ENABLE_VMM: 1
  GGML_SYCL_PRIORITIZE_DMMV: 0
  GGML_SYCL_USE_ASYNC_MEM_OP: 1
  GGML_SYCL_ENABLE_FLASH_ATTN: 1
  GGML_SYCL_USM_SYSTEM: 0
Found 1 SYCL devices:
|  |                   |                                       |       |Max    |        |Max  |Global |                     |
|  |                   |                                       |       |compute|Max work|sub  |mem    |                     |
|ID|        Device Type|                                   Name|Version|units  |group   |group|size   |       Driver version|
|--|-------------------|---------------------------------------|-------|-------|--------|-----|-------|---------------------|
| 0| [level_zero:gpu:0]|             Intel Arc Pro B70 Graphics|   20.2|    256|    1024|   32| 33456M|           1.15.37858|
SYCL Optimization Feature:
|ID|        Device Type|Reorder|
|--|-------------------|-------|
| 0| [level_zero:gpu:0]|      Y|
Backend 1/2: SYCL0
  Device description: Intel(R) Arc(TM) Pro B70 Graphics
  Device memory: 31906 MB (31906 MB free)

  [FLASH_ATTN_EXT] ERR = 0.014243019 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,1],kv=1024,nb=32,mask=1,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.153135469 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=32,mask=1,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.264602165 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=75,mask=1,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.284017350 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.266853857 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=75,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 0.000806602 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,1],kv=1024,nb=32,mask=0,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 0.013846743 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,1],kv=1024,nb=75,mask=0,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.157263827 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=32,mask=0,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.477912927 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=75,mask=0,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.211760724 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=32,mask=0,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
[FLASH_ATTN_EXT] ERR = 1.434984904 > 0.000500000   FLASH_ATTN_EXT(hsk=40,hsv=40,nh=4,nr23=[1,3],kv=1024,nb=75,mask=0,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3]): ←[1;31mFAIL←[0m
  FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=113,nb=1,mask=1,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=iq4_nl,type_V=iq4_nl,permute=[0,1,2,3]): not supported [SYCL0]
  FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=113,nb=3,mask=1,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=iq4_nl,type_V=iq4_nl,permute=[0,1,2,3]): not supported [SYCL0]
  FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=113,nb=32,mask=1,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=iq4_nl,type_V=iq4_nl,permute=[0,1,2,3]): not supported [SYCL0]
  FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=113,nb=75,mask=1,sinks=1,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=iq4_nl,type_V=iq4_nl,permute=[0,1,2,3]): not supported [SYCL0]
(DELETED ALL THE OK ONES)

G:\Projects\AI\llama-sycl-build-25312\ggml\src\ggml-sycl\fattn.cpp:92: Not match KV type in vec
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |          pp8192 |        759.23 ± 0.33 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |           tg128 |         20.55 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |  pp8192 @ d8192 |        527.47 ± 0.84 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |   tg128 @ d8192 |         18.99 ± 0.01 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 | pp8192 @ d65536 |        233.70 ± 0.19 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | SYCL       |  99 |   1 |  tg128 @ d65536 |         12.50 ± 0.01 |

build: 13f503255 (9845)

johnkarlhill · 2026-07-07T18:48:50Z

@SleepinDevil After running countless numbers of benchmarks, I've stopped adding the ARCH flag. I am using "cmake --preset x64-windows-sycl-release -DGGML_SYCL_F16=ON -DGGML_SYCL_DNN=ON" and found it to be almost as performant.

I am running this PR as my production build so I know the performance is there, especially with 27B. I just need to clean my build directory after finishing up the MKL bugs found today... Many of those bug fixes need to be added to this PR. Give me a couple days and I'll perform a fresh build to see if the performance holds.

SleepinDevil · 2026-07-07T22:47:19Z

@johnkarlhill will keep an eye out for that :). Yes, I don't know why the FA backend ops test shows failures in mine too.

I also have no idea how I managed to have two versions of the oneAPI Deep Learning Essentials installed, I need to do a cleanup of my environment first I think and then try again 😅. I'll try without the arch flag too.

Thanks for your efforts though, looking forward to more performance being unlocked from the Intel B70s, I cannot believe how good they are genuinely.

simonlui · 2026-07-09T18:07:01Z

This seems to be bugged with Alchemist based cards. I tried merging and running this on my A770 and ran the test-backend-ops and got the following outputs that failed.

[FLASH_ATTN_EXT] ERR = 1.774325900 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 2.109317422 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.924418137 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=1024,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.895588113 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=1024,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.918028822 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.985922848 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.931638425 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=1024,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.983022251 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=1024,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.971164877 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.936653299 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.920868213 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=1024,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.892099531 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=1024,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.945075866 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=2048,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.864397727 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=2048,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.812062676 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=2048,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.931935803 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=8,nr23=[4,1],kv=2048,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.886857950 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=2048,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.956302377 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=2048,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.992294613 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=2048,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.873725479 > 0.000500000   FLASH_ATTN_EXT(hsk=128,hsv=128,nh=8,nr23=[4,1],kv=2048,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.931195716 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=2048,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.911365922 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=2048,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.900233276 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=2048,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.964257456 > 0.000500000   FLASH_ATTN_EXT(hsk=256,hsv=256,nh=8,nr23=[4,1],kv=2048,nb=64,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.869623285 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f32,type_V=f32,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.824458223 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=bf16,type_V=bf16,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.756930950 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.777304532 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q5_1,type_V=q5_1,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 1.829089712 > 0.000500000   FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q5_0,type_V=q5_0,permute=[0,1,2,3]): FAIL
[FLASH_ATTN_EXT] ERR = 2.038435397 > 0.000500000 FLASH_ATTN_EXT(hsk=64,hsv=64,nh=4,nr23=[1,1],kv=1024,nb=32,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_1,type_V=q4_1,permute=[0,1,2,3]): FAIL

I know it is still a WIP but just letting you know this is the case.

johnkarlhill · 2026-07-09T20:25:02Z

@simonlui It was recently discovered that the oneDNN releases right now are slightly broken for Alchemist cards. I highly suggest sticking with PR25025 for prefill performance increases as MKL does not use oneDNN. I plan on acquiring an Alchemeist card for testing in the future so, hopefully, something can be done to get the extra SPDA performance for Alchemist.

vpirogov · 2026-07-10T03:19:53Z

@simonlui It was recently discovered that the oneDNN releases right now are slightly broken for Alchemist cards.

Based on investigation in uxlfoundation/oneDNN#5510 there's a bug in latest version of the GPU driver for Alchemist that causes the failure. Using previous version works as expected.

KellenRenshaw · 2026-07-12T21:33:21Z

@johnkarlhill I have an unused Alchemist card I can send you if you are interested. Its a Sparkle A770-16GB card. Let me know if you are interested and we can arrange shipping.

NateHag · 2026-07-15T00:05:22Z

Tested this PR on Battlemage workstation hardware. The oneDNN SDPA path delivers a large prefill win with quantized KV. I initially hit GPU hangs, but after a long bisection they turned out to be unrelated to this PR: they reproduce on stock master and are caused by an interaction between the Level Zero v2 adapter and the GGML SYCL VMM pool. Details and workarounds below since anyone benching this PR with quantized KV will probably hit them.

Setup

GPU: Intel Arc Pro B70, xe driver, kernel 6.17.0-1028-oem
Level Zero V2 1.15.38646, NEO 26.22.38646.6
oneAPI 2026.0 (icpx), oneDNN 2026.0.1, -DGGML_SYCL=ON -DGGML_SYCL_TARGET=INTEL -DGGML_SYCL_F16=ON -DGGML_SYCL_DNN=ON
PR head 9e9524e, compared against master a4ce259 built with identical flags
Model: Qwen2 7B Q4_K_M

Performance (llama-bench, t/s, r=3, all with the VMM workaround below)

config	pp512	pp2048	pp512 d4096	pp2048 d4096	tg32 d4096
master, fa=off, f16 KV	3387	2888	2019	1944	74.0
master, fa=on, f16 KV	3080	2470	1313	1180	70.3
master, fa=on, q8_0 KV	3000	2426	1304	1169	61.3
PR, fa=on, f16 KV	3150	2689	1641	1468	70.4
PR, fa=on, q8_0 KV	3099	3455	3130	3054	60.9
PR, fa=on, q4_0 KV	3110	3460	3221	3211	61.6

Prefill at depth with quantized KV is about 2.6x master and clearly worth it. Tier isolation with your env switches confirmed the gain is the HYBRID oneDNN path: 3230 t/s hybrid vs 1667 MKL vs 1330 tile at pp512 d4096.

Under a sustained concurrency test (llama-server, 8 slots, 32 concurrent clients, mixed prompt lengths, 3 minutes) the PR sustains the load fine once the environment issue below is worked around: zero timeouts, overload recovery in under 100 ms, flat throughput over a soak.

Environment issue you may hit while testing: quantized KV FA hangs the ccs engine (also on master)

Symptoms: llama-bench with -fa 1 -ctk q4_0 -ctv q4_0 occasionally wedges for good, and llama-server with q8_0 KV plus FA reliably wedges within minutes under concurrent load. dmesg shows xe GT0: Engine reset: engine_class=ccs and a device coredump. The server keeps answering /health but every completion hangs.

Bisection results:

Reproduces on stock master a4ce259, so it is not introduced by this PR
Only triggers when FA reads a quantized KV cache (f16 KV is clean; K-only quantization without FA is clean)
Requires mixed prefill plus decode traffic; prefill-only and decode-only loads are clean
Adding a stream->wait() after every FLASH_ATTN_EXT op makes it vanish (timing dependent)
GGML_SYCL_ENABLE_VMM=0 makes it vanish completely at identical throughput
UR_LOADER_USE_LEVEL_ZERO_V2=0 (legacy adapter) also makes it vanish completely

So it looks like a Level Zero v2 immediate-command-list adapter issue with allocations from the GGML VMM pool (reserve/map) being touched by in-flight kernels, not a llama.cpp logic bug. I will file it against oneAPI/UR separately. Until then, anyone testing this PR with quantized KV on Battlemage probably wants GGML_SYCL_ENABLE_VMM=0, and it may be worth a note in the PR description.

Happy to rerun with patches or provide the xe devcoredump and full bisection logs.

NateHag · 2026-07-15T00:33:34Z

Correction to my comment above: GGML_SYCL_ENABLE_VMM=0 reduces the hang frequency but does not eliminate it. It survived several 3 to 4 minute concurrency runs (which is what I based the claim on), but a 30 minute soak at ~80 percent capacity wedged again after roughly 28 minutes with the same ccs engine reset signature, quantized KV plus FA, VMM pool disabled.

So the failure interval moved from minutes to tens of minutes, which is consistent with a low probability race rather than a fixed allocation pattern. Treat the VMM setting as mitigation, not a fix. I am currently soaking the legacy adapter path (UR_LOADER_USE_LEVEL_ZERO_V2=0) for the same 30 minutes and will report the result either way. Sorry for the noise, the short runs looked deceptively clean.

NateHag · 2026-07-15T01:13:26Z

Follow-up as promised: the legacy adapter path (UR_LOADER_USE_LEVEL_ZERO_V2=0, combined with GGML_SYCL_ENABLE_VMM=0) also fails the 30 minute soak. It ran 8 clean minutes at ~80 percent capacity, then hit the same ccs engine reset; this time the server process aborted with UR_RESULT_ERROR_DEVICE_LOST on the next synchronize.

So the current picture for Battlemage (B70, xe, NEO 26.22.38646.6, Level Zero 1.15.38646):

FA over a both-quantized KV cache under sustained mixed prefill+decode load hangs the compute engine on stock master and on this PR alike; f16 KV is immune.
Neither disabling the VMM pool nor the legacy adapter fixes it; they only stretch the mean time to failure from minutes to tens of minutes. Short benchmarks (including several minutes of concurrency) look deceptively clean.
A host sync after every FLASH_ATTN_EXT op eliminates it, so it smells like a missing dependency/residency edge between enqueued work and pool-recycled or freshly mapped allocations, somewhere below the SYCL runtime.

The perf numbers in my first comment stand (measured between hangs, and prefill-only runs are stable). Happy to run instrumented builds, bisect driver versions, or ship the xe devcoredumps to whoever picks this up.

NateHag · 2026-07-15T19:13:25Z

Platform follow-up on the hangs discussed in my comments above (filed as #25692): I moved the same B70 card to a different host — Ubuntu 26.04, kernel 7.0.0-27 (newer xe), card in an external OCuLink dock (link trained Gen3 x4), NEO 26.22 — and could not reproduce any xe instability under sustained load. Three clean-boot 30-minute server soaks (different serving stack: vLLM-XPU 0.25.1 with an AWQ int4 model, continuous batching, including one run pinned at full prefill saturation for the entire 30 minutes) completed with zero engine resets and zero devcoredumps. On the original host (kernel 6.17.0-1028-oem, direct PCIe slot) this same card reproducibly reset within minutes — including under workloads unrelated to this PR.

Caveats: this is not a rerun of the FA + quantized-KV config from this PR, and two variables changed at once (kernel 6.17→7.0 xe, and slot → OCuLink x4 link path), so it does not identify the failing layer. But the same silicon running heavy sustained compute cleanly on a newer kernel makes a kernel-xe/platform contribution to the reset family look more likely than a card defect. Will report back if I isolate kernel vs link path.

johnkarlhill · 2026-07-17T02:46:15Z

@johnkarlhill I have an unused Alchemist card I can send you if you are interested. Its a Sparkle A770-16GB card. Let me know if you are interested and we can arrange shipping.

That is awesome of you to suggest but I would feel horrible if if something happened to it or, if for some reason, I couldn't get the performance out of it that I'm hoping for. Thank you but I must decline.

johnkarlhill · 2026-07-18T20:44:59Z

@KellenRenshaw I'm not giving up on Alchemist. I think PR25025 will give you a nice performance jump over stock but it's not merged yet. As soon as I get the funds to purchase an Alchemist card, I'll dig in to see what needs to happen.

johnkarlhill · 2026-07-18T20:45:18Z

@hmscider @simonlui @SleepinDevil @KellenRenshaw @NateHag @vpirogov

Closing this PR to cut a cleaner version. The original "hybrid" approach depended on #25025 (MKL GEMM), which hasnt merged yet. The new PR strips the MKL dependency entirely and feeds dequantized K/V directly into oneDNN SDPA:

#25874

All feedback from this thread informed the rewrite. Thanks for the input.

johnkarlhill and others added 8 commits June 28, 2026 18:23

SYCL: add oneMKL GEMM flash attention for XMX-accelerated prompt proc…

763c40f

…essing

fattn-mkl: rename mkl_disable -> mkl_enable for clarity

e8b207e

johnkarlhill requested a review from a team as a code owner July 5, 2026 04:25

github-actions Bot added documentation Improvements or additions to documentation ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Jul 5, 2026

johnkarlhill force-pushed the sycl-fa-frankenmerge branch from e0b84bd to 9e9524e Compare July 6, 2026 01:18

johnkarlhill mentioned this pull request Jul 6, 2026

[SYCL] Flash Attention with XMX engine via oneDNN graph API (SDPA) on KV f16 for Xe2 ; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k #25222

Merged

johnkarlhill marked this pull request as draft July 6, 2026 14:16

hybrid: port seq-view stride fix from MKL, env-var one-liner

66993fd

NateHag mentioned this pull request Jul 15, 2026

SYCL: GPU hang (xe ccs engine reset) with flash attention + quantized KV cache under sustained server load on Arc Pro B70 #25692

Open

johnkarlhill closed this Jul 17, 2026

johnkarlhill deleted the sycl-fa-frankenmerge branch July 17, 2026 02:50

johnkarlhill reopened this Jul 17, 2026

johnkarlhill closed this Jul 18, 2026

johnkarlhill changed the title ~~SYCL: oneDNN SDPA Flash Attention for Quantized KV Caches~~ Extended SYCL oneDNN SDPA to non-FP16 KV caches (Q4_0-Q8_0 and FP32) Jul 18, 2026

Uh oh!

Conversation

johnkarlhill commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggml-gh-bot Bot commented Jul 5, 2026

Uh oh!

hmscider commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SleepinDevil commented Jul 5, 2026

Uh oh!

johnkarlhill commented Jul 5, 2026

Uh oh!

johnkarlhill commented Jul 5, 2026

Uh oh!

SleepinDevil commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnkarlhill commented Jul 7, 2026

Uh oh!

SleepinDevil commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonlui commented Jul 9, 2026

Uh oh!

johnkarlhill commented Jul 9, 2026

Uh oh!

vpirogov commented Jul 10, 2026

Uh oh!

KellenRenshaw commented Jul 12, 2026

Uh oh!

NateHag commented Jul 15, 2026

Uh oh!

NateHag commented Jul 15, 2026

Uh oh!

NateHag commented Jul 15, 2026

Uh oh!

NateHag commented Jul 15, 2026

Uh oh!

johnkarlhill commented Jul 17, 2026

Uh oh!

johnkarlhill commented Jul 18, 2026

Uh oh!

johnkarlhill commented Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

johnkarlhill commented Jul 5, 2026 •

edited

Loading

hmscider commented Jul 5, 2026 •

edited

Loading

SleepinDevil commented Jul 7, 2026 •

edited

Loading

SleepinDevil commented Jul 7, 2026 •

edited

Loading