Misc. bug: SYCL: bad performance on newer oneAPI

[WizardlyBump17]

Name and Version

root@9c90ac50a126:/app# ./llama-cli --version
version: 10074 (2beefef68)
built with IntelLLVM 2026.1.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

No response

Command line

`./llama-server --host 0.0.0.0 --port 8080 --model /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf.ignore --jinja --threads 8 --ctx-size 262144 --cache-ram 0 --parallel 1 --temperature 0.0 --top-p 0.2 --top-k 20 --no-mmap --spec-type draft-mtp --spec-draft-n-max 3 --batch-size 2700 --ubatch-size 2700 --n-gpu-layers 99 --n-cpu-moe 99 --ctx-checkpoints 0 --no-cache-prompt`

`podman build --tag=test2 --target=full --file=.devops/intel.Dockerfile --build-arg=IGC_VERSION=v2.38.2 --build-arg=IGC_VERSION_FULL=2_2.38.2+22051 --build-arg=COMPUTE_RUNTIME_VERSION=26.27.39122.11 --build-arg=COMPUTE_RUNTIME_VERSION_FULL=26.27.39122.11-0 --build-arg=IGDGMM_VERSION=22.10.0 --build-arg=ONEAPI_VERSION=2026.1.1-devel-ubuntu24.04 .`

Problem description & steps to reproduce

When upgrading to newer oneAPI (--build-arg=ONEAPI_VERSION=2026.1.1-devel-ubuntu24.04) the PP performance drops by a lot. #25222 brings a very good performance uplift, but when you use newer oneAPI, that performance is blown as if that Pull Request never existed.

First Bad Commit

No response

Relevant log output

No response

[DanRRRR]

May be this is not the OneAPI. I had drop of performance last two days by the factor of 3. And for everything, including even the model load. Searching for 2 days, tried all the options, changed BLAS, quantization, AVX512, numa isolation, numactl, number of threads on dual socket system, reboot, full new git clone.... How many t/s do you get?

[WizardlyBump17]

May be this is not the OneAPI. I had drop of performance last two days by the factor of 3. And for everything, including even the model load. Searching for 2 days, tried all the options, changed BLAS, quantization, AVX512, numa isolation, numactl, number of threads on dual socket system, reboot, full new git clone.... How many t/s do you get?

It is oneAPI. I had this issue for a while, but I thought maybe newer oneAPI would fix it eventually, but it didnt

[NeoZhangJianyu]

export DNNL_VERBOSE=1

it will show the oneDNN OP performance data in ms.
By comparing with the data of old oneAPI, we will know which oneDNN OP is regression.
Copy/paste the oneDNN, we can report issue to oneDNN team.

Thank you!

[vpirogov]

There's a known regression in oneDNN FlashAttention with IGC 2.36. Does reverting IGC to 2.32 or something help?

[WizardlyBump17]

There's a known regression in oneDNN FlashAttention with IGC 2.36. Does reverting IGC to 2.32 or something help?

I rolled back to https://github.com/intel/compute-runtime/releases/tag/26.14.37833.4 and got no luck. The performance is still bad

[WizardlyBump17]

export DNNL_VERBOSE=1

it will show the oneDNN OP performance data in ms. By comparing with the data of old oneAPI, we will know which oneDNN OP is regression. Copy/paste the oneDNN, we can report issue to oneDNN team.

Thank you!

That did not generate any extra logs

[vpirogov]

That did not generate any extra logs

Ok then, oneDNN is off the hook on this one :)

[gdaswani]

The latest builds are quite unstable, been trying it with ONEAPI=2026.1.1-devel-ubuntu26.04 - can't even get past a round of llama-bench benchmarks. After a while, it just starts returning jibberish in the chat interface. No problems when using the llama-cpp vulkan back end, and intel's llm-scaler is working as well. Pretty easy to replicate on ubuntu 26.04.

I build it using

docker build -t llamacpp-sycl-b70:local .

I compose and run it by

docker compose up -d

---------- Docker -------------

ARG ONEAPI=2026.1.0-devel-ubuntu26.04

FROM intel/oneapi-toolkit:${ONEAPI} AS build
RUN apt-get update && apt-get install -y --no-install-recommends \
      git cmake libssl-dev libcurl4-openssl-dev ca-certificates \
 && rm -rf /var/lib/apt/lists/*
RUN git clone --depth=1 https://github.com/ggml-org/llama.cpp.git /src
WORKDIR /src
RUN . /opt/intel/oneapi/setvars.sh --force >/dev/null 2>&1 || true && \
    cmake -B build -DGGML_SYCL_F16=ON -DGGML_NATIVE=OFF -DGGML_SYCL=ON -DBUILD_SHARED_LIBS=OFF \
      -DLLAMA_OPENSSL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx && \
    cmake --build build --config Release -j"$(nproc)" && \
    mkdir -p /opt/llama/bin && cp build/bin/llama-* /opt/llama/bin/

FROM intel/oneapi-toolkit:${ONEAPI} AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
      curl gpg ca-certificates libcurl4 \
 && curl -fsSL https://repositories.intel.com/gpu/intel-graphics.key \
      | gpg --dearmor -o /usr/share/keyrings/intel-graphics.gpg \
 && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu noble unified" \
      > /etc/apt/sources.list.d/intel-gpu.list \
 && apt-get update && apt-get install -y --no-install-recommends \
      libze-intel-gpu1 libze1 \
 && rm -rf /var/lib/apt/lists/*
COPY --from=build /opt/llama/bin/ /usr/local/bin/
EXPOSE 8080
ENTRYPOINT ["/bin/bash","-c","source /opt/intel/oneapi/setvars.sh --force >/dev/null 2>&1 || true; exec stdbuf -oL -eL llama-server \"$@\"","--"]

---------- compose.yaml ------------------

services:
  llama-server-sycl:
    image: llamacpp-sycl-b70:local
    container_name: llama-cpp-sycl
    ports: ["8080:8080"]
    devices: ["/dev/dri:/dev/dri"]
    group_add: ["video", "991"]   # your render-node GID; check `ls -l /dev/dri`
    ipc: host
    volumes: 
      - llama-cache:/root/.cache
      - type: bind
        source: /storage/docker-volumes/llm/models
        target: /models
    environment:
      - GGML_SYCL_USE_LEVEL_ZERO_API:1
      - ONEAPI_DEVICE_SELECTOR=level_zero:0
      - UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1
      - SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
      - ZES_ENABLE_SYSMAN=1
    command:
      - "-m"
      - "/models/Qwen/Qwen3.6-27B-UD-Q4_K_XL.gguf"
      - "--threads"
      - "1"
      - "-c"
      - "131072"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8080"
      - "--temp"
      - "0.6"
      - "--top-p"
      - "0.95"
      - "--top-k"
      - "20"
      - "--min-p"
      - "0.0"
      - "--presence-penalty"
      - "0.0"
      - "--repeat-penalty"
      - "1.0"
      - "--flash-attn"
      - "on"
      - "--n-gpu-layers"
      - "-1"
      - "--spec-type"
      - "draft-mtp"
      - "--spec-draft-n-max"
      - "2"
      - "--reasoning"
      - "on"
      - "--no-mmap"
      - "--jinja"
      - "--chat-template-kwargs"
      - "{\"preserve_thinking\": true}"
    restart: unless-stopped
    deploy:
        resources:
            limits:
                cpus: '1.0'
volumes:
  llama-cache:

[WizardlyBump17]

I just tried running apt-get upgrade inside the built container and while there was a little performance regression (462 -> 455), nothing much changed. The issue seems to be when compiling