Skip to content

[AMDGPU] Fix cross-half shuffle select for top-half lanes on wave64 #769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ZJLi2013 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[AMDGPU] Fix cross-half shuffle select for top-half lanes on wave64 #769

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions quadrants/runtime/llvm/runtime_module/runtime.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1096,7 +1096,6 @@ i32 amdgpu_cross_half_shuffle_i32(i32 target_lane, i32 value) {
// cross-SIMD case via the ``swapped`` payload. On CDNA the wave is one SIMD64 so both reads return the same value
// and the select is a no-op; we don't try to optimize that out because the dead read is cheap (LLVM CSE may fold
// it anyway).
i32 self_lane = amdgpu_lane_id();
i32 swapped = amdgpu_permlane64(value);
i32 byte = (target_lane & 31) * 4;
// ``llvm.amdgcn.ds.bpermute`` is the real hardware ``ds_bpermute_b32`` -- but if LLVM's uniformity analysis decides
Expand All @@ -1122,9 +1121,16 @@ i32 amdgpu_cross_half_shuffle_i32(i32 target_lane, i32 value) {
#if defined(ARCH_amdgpu) && (defined(__x86_64__) || defined(__i386__) || defined(__amdgcn__))
__asm__ volatile("" : "+v"(byte));
#endif
// ``byte`` masks the lane index to 5 bits, so ``from_self_half`` always reads the low SIMD32
// (lanes 0..31) and ``from_other_half`` (via the ``permlane64`` swap) always reads the high SIMD32
// (lanes 32..63) -- regardless of which half the executing lane sits in. The correct payload is
// therefore selected purely by which half the *target* lane lives in (``target_lane & 32``); it must
// not depend on the executing lane. The previous ``(target_lane ^ self_lane) & 32`` was wrong for
// top-half lanes: for a same-half target it wrongly picked the other-half read, so e.g.
// ``shuffle_up`` on lanes 32..63 returned lane ``target_lane - 32`` instead of ``target_lane``.
i32 from_self_half = amdgpu_ds_bpermute(byte, value);
i32 from_other_half = amdgpu_ds_bpermute(byte, swapped);
return ((target_lane ^ self_lane) & 32) ? from_other_half : from_self_half;
return (target_lane & 32) ? from_other_half : from_self_half;

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 9, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Gate target-half selection on RDNA generation

On gfx10/RDNA1-2, which this backend still supports via the has_permlane64 == false LDS fallback in llvm_context.cpp, ds_bpermute in wave64 operates within each 32-lane half rather than as a wave-global low-half read. With this unconditional target-half select, a top-half lane doing an in-half read (for example identity shuffle with self_lane == target_lane == 40) selects from_other_half, so it reads the swapped payload for lane 8 instead of lane 40; top-half cross-half reads are inverted as well. This needs to keep the self-relative selection for those targets or be gated by the actual ds_bpermute semantics.

Useful? React with 👍 / 👎.

@hughperkins hughperkins Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ZJLi2013 thoughts on codex comment?

}

i32 amdgpu_shuffle_i32(i32 index, i32 value) {
Expand Down