Skip to content

sycl: single-device still requires sync #25741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
malsbat wants to merge 1 commit into from
Closed

sycl: single-device still requires sync #25741

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 5 additions & 10 deletions ggml/src/ggml-sycl/fattn-onednn.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,16 +245,11 @@ void ggml_sycl_flash_attn_ext_onednn(ggml_backend_sycl_context & ctx, ggml_tenso
E.cp.execute(strm, ti, {to});

permute_sdpa_out_sycl(outf.get(), (float *) dst->data, mb, H, q, d, stream);
// Single device: no sync is required, and actually PP perf is ~6% > wait_and_throw() (tested on llama-3.1-8b & qwen3.6-27b, both Q8_0, with Arc B70).
// Any future multi-GPU refactor MUST re-measure this single-device path and keep the best
// single-device PP speed. Otherwise (multiple devices/streams can race the reuse):
if (ggml_sycl_info().device_count > 1) {
// cont_to_f16 -> oneDNN execute -> permute is async on this stream, but the
// pool_alloc*s above free their device buffers at host return. Without this wait the next
// scheduler op re-acquires those bytes while the GPU is still computing the SDPA, turning
// it into garbage and collapsing multi-turn output to a single repeated token ("GGGGG...").
stream->wait_and_throw();
}
// cont_to_f16 -> oneDNN execute -> permute is async on this stream, but the
// pool_alloc*s above free their device buffers at host return. Without this wait the next
// scheduler op re-acquires those bytes while the GPU is still computing the SDPA, turning
// it into garbage and collapsing multi-turn output to a single repeated token ("GGGGG...").
stream->wait_and_throw();
}
catch (const std::exception & e) {
// any oneDNN/SYCL failure is non-fatal: fall back to the existing kernel (strictly additive).
Expand Down