[AMDGPU] Keep fdiv correctly rounded under fast_math (fixes modulo bug #749)#773
Open
ZJLi2013 wants to merge 1 commit into
Open
[AMDGPU] Keep fdiv correctly rounded under fast_math (fixes modulo bug #749)#773ZJLi2013 wants to merge 1 commit into
ZJLi2013 wants to merge 1 commit into
Conversation
Under fast_math the global IRBuilder fast-math flags include afn (ApproxFunc). The AMDGPU backend uses that to lower fdiv to an approximate v_rcp_f32 reciprocal (~2.5 ULP), so exact divisions such as 180.0/180.0 return 0.99999994. This breaks floor(a/b) and the a - floor(a/b)*b modulo expansion, surfacing as the "AMDGPU modulo bug" where float % (runtime int) returns the unreduced dividend. Clear afn on the emitted FDiv so division stays correctly rounded (matching CPU/CUDA, where it is a no-op). afn is intended for transcendentals, not basic arithmetic. Fixes: quadrants#749 Co-authored-by: Cursor <cursoragent@cursor.com>
Collaborator
|
Could we gate this change on using AMD please? That would make me feel more comfortable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #749
Summary
On AMDGPU,
float % (int loaded from a field)returns the unreduced dividend instead of the remainder (reported in #749 as the "AMDGPU modulo bug"). The root cause is not modulo-specific: AMDGPU float division is being lowered to an approximate reciprocal, so exactly-divisible cases like180.0 / 180.0return0.99999994. Anyfloor(a / b)— spatial hashing, grid indexing (support_res), bucketization — silently lands one cell off. Modulo just makes it obvious because it is lowered asa - floor(a/b)*b.Fix: clear the
afn(ApproxFunc) fast-math flag for the emittedFDivso division stays correctly rounded.afnis meant for transcendentals, not basic arithmetic; the change is a no-op on CPU/CUDA (theirfdivis already correctly rounded).Root cause
compile_config.fast_mathdefaults totrue.TaskCodeGenLLVM::initialize_context()sets global fast-math flags on the IRBuilder, includingsetApproxFunc():Every instruction, including
BinaryOpType::div→CreateFDiv, inheritsafn. The AMDGPU backend usesafnonfdivto select the fastv_rcp_f32-based reciprocal path (~2.5 ULP) instead of a correctly-rounded division. x86 and NVPTX ignoreafnforfdiv(their hardware divide is correctly rounded regardless), so the bug is AMDGPU-only.Why
float % literallooks fine butfloat % fielddoes not: a constant divisor is folded at compile time (constant_fold.cpp, exact), while a runtime field value takes thefdivpath and hits the approximate reciprocal.Trace of
floor(180.0) % mod[None](mod=180), storing every intermediate as f32:x / mf(float / field-int)0.999999941.0floor(x / mf)0.01.0floor(x / const_180)1.01.0x - floor(x/mf)*mf(mod)180.00.0Setting
fast_math=Falsemakesfloor(x/mf) = 1.0andmod = 0on AMDGPU, confirming the trigger isafn-on-fdiv.Fix
In
TaskCodeGenLLVM::visit(BinaryOpStmt)forBinaryOpType::div(real branch), clearafnfor the emittedFDivvia aFastMathFlagGuard, leaving the builder's other fast-math flags untouched.modandfloordivare lowered toBinaryOpType::divindemote_operations.cpp, so this one spot coversfloat / int,floordiv, and%alike.Two non-obvious points worth flagging for review:
CreateFDivbuilds the instruction). A post-hocinst->setFastMathFlags(...)afterCreateFDivdid not take effect on AMDGPU in practice.180.0 / field, before → after:Testing
Standalone quadrants reproducer from #749 (
qd_modulo_bug.py, cases A–F) plus a step-by-step intermediate-value diagnostic. Built from source and validated on:main@2e2b43b40)float % int-literalfloat % int-from-fieldint(float) % fieldfloor(180.0 / field_180)No change to non-
floorfloat results;fast_mathapproximation for transcendentals (sin/cos/exp/…) is retained.Notes / scope
fdivalready ignoresafn). Clearing the flag keeps behavior consistent across backends.