sycl: parallelize the non-contiguous concat kernel#25852
Conversation
github-actions
Bot
added
ggml
There was a problem hiding this comment.
For Qwen3.6-27B-UD-Q5_K_XL-MTP.gguf on B60.
I find the performance is reduced.
Through the LLM is not the target of this PR, it won't be impacted negative.
| Test | fa | Base t/s | Primary t/s | Increase Rate (Primary vs Base) |
|---|---|---|---|---|
| pp512 | 0 | 44.87 | 41.28 | -8.00% |
| pp512 | 1 | 41.29 | 38.04 | -7.87% |
| tg128 | 0 | 2.75 | 2.49 | -9.45% |
| tg128 | 1 | 2.77 | 2.50 | -9.75% |
Of course, I will test with Qwen3.6-27B-UD-Q4_K_XL.gguf.
Could you share the whole cmd to show the benefit of this PR?
Launch geometry only: the non-contiguous concat kernel launched a single-lane work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one. SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`. llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV), on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)
0553bf6 to
e792674
Compare
|
@arthw It seems that you are offloading the model to system memory? Your tg128 is super low! The command I used to get benchmarks was: I do not have a B60 to test, and I cannot repro the regression on my hardware. I believe it is due to oversubscribing the hardware when there isn't enough elements to fill up SYCL_CONCAT_BLOCK_SIZE. I just pushed a commit that may address your issue. |
|
With the latest update in the PR and Qwen3.6-27B-UD-Q4_K_XL.gguf on B60. ./build/bin/llama-bench -m ../models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 0,1
./build/bin/llama-bench -m ../models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1 -ctk q8_0 -ctv q8_0 -p 512,2048 -n 128 A good performance optimization shouldn't reduce other cases' perf when increase the target case. Please check it! Thank you! |
…ONCAT_BLOCK_SIZE)
|
@arthw I still cannot reproduce the performance regression on my hardware. Could you try different values of this new environment variable I added: |
|
@Titaniumtown What's your GPU and driver version? |
|
I'm on kernel 7.0.14 on NixOS GPU: Intel Arc Pro B70 Graphics: PCI 8086:E223 (subsys 8086:1701) |
|
@Titaniumtown Here is the test result on B60 with Qwen3.6-27B-UD-Q4_K_XL.gguf case 1Base: ./build/bin/llama-bench -m ..//models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1 -ctk q8_0 -ctv q8_0 -p 512,2048 -n 128
PR:
build: 2709909 (10068) export GGML_SYCL_CONCAT_BLOCK_SIZE=128
build: 2709909 (10068) export GGML_SYCL_CONCAT_BLOCK_SIZE=256
case 2./build/bin/llama-bench -m ..//models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1 base
PR:
build: 2709909 (10068) export GGML_SYCL_CONCAT_BLOCK_SIZE=128
build: 2709909 (10068) export GGML_SYCL_CONCAT_BLOCK_SIZE=256
|
|
@arthw so your performance regression went away? Interesting. Maybe was transient? |
|
Yes, the latest code increase the performance in the test. |
|
Glad to hear it! Let me know if there's anything else I can do for this PR! |
2 participants
Overview
The non-contiguous concat kernel launched a single-lane work-group (1, 1, 1), this PR changes it so that it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one that can better utilize the hardware instead of having single-width iteration loops.
llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV), on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)
Additional information
Should the 256 value that is shared among a bunch of macros in ggml/src/ggml-sycl/presets.hpp be dynamically set similar to #25205? I am wondering what the performance impact of setting that to 128 on Intel Alchemist hardware would be? May be faster due to 128 being the native work-group size.
Requirements