celt: complex IMDCT/MDCT archsimd kernels (needs the Reverse64 op) - #488
Merged
Conversation
…4 patch) Port the IMDCT pre-rotation and TDAC windowing kernels to archsimd, gated goexperiment.simd && !purego. Both rely on a full 4-lane reverse (reverse4) built on Float32x4.Reverse64 — the arm64 VREV64 op added in github.com/thesyncim/go branch arm64-simd-reverse64 — for their descending spectrum/window accesses; they will not build on a vanilla gotip. Bit-exact vs the scalar references (TestIMDCTPreRotateFMA32KissMatchesScalar, TestIMDCTTDACWindowFMA32MatchesScalar). imdct_tdac (real-valued, reverse only) beats the hand asm on M4 Max (-15%); imdct_pre (complex re/im deinterleave) is +26%, the cost of archsimd lacking a VLD2-style deinterleaving load. The asm/purego paths stay the default for non-experiment builds.
Port the IMDCT post-rotation (imdctPostRotateF32FromKiss) from hand asm to Go simd/archsimd, gated goexperiment.simd && !purego (needs the Reverse64 op from the patched toolchain). Forward four come from a ConcatEven/Odd deinterleave of the kissCpx scratch; the backward four walk down and reverse4 so descending lanes line up; products are single-round Muls and accumulates fused MulAdds, matching mdctMulAddMix/mdctMulSubMix bit-for-bit. The asm .go/.s retag !goexperiment.simd. Benchmarks at asm parity.
Port the forward-MDCT post-twiddle (mdctPostTwiddleNeon) from hand asm to Go simd/archsimd, gated goexperiment.simd && !purego. Each block pairs a forward run and its mirror j=n4-1-i; ConcatEven/Odd deinterleaves the kissCpx, the two ends tile coeffs contiguously via reverse4 + InterleaveLo/Hi. Products are single-round Muls and the combines plain Sub/Add (no fusion), matching mdctMul and the scalar loop bit-for-bit. The asm .go/.s retag !goexperiment.simd.
Port the forward-MDCT middle fold (mdctMidFoldStoreNeon) from hand asm to Go simd/archsimd, gated goexperiment.simd && !purego. The (re,im) compute is vectorized four lanes at a time (ConcatEven deinterleave, descending re via reverse4, fused MulAdd, scaling Mul) matching mdctStoreDirectStageFMALike bit-for-bit; the bit-reversed dst[bitrev[i]] scatter stays scalar, exactly as the asm does. The asm .go/.s retag !goexperiment.simd.
Port the two windowed forward-MDCT fold kernels (mdctFold1StoreNeon, mdctFold3StoreNeon) from hand asm to Go simd/archsimd, gated goexperiment.simd && !purego. Each deinterleaves six sample/window streams (ConcatEven ascending, reverse4 descending) and combines them with fused MulAdds + single-round Muls matching mdctMulAddMixEncode/mdctMulSubMixEncode/ mdctMulSubMixAlt; the shared twiddle/scale/bit-reversed scatter tail (mdctFoldStore) mirrors mid_fold. The asm .go/.s retag !goexperiment.simd.
Owner
Author
|
Consolidated into #487 — single PR for the whole archsimd kernel migration (vanilla + silk widen + complex, the complex path gated behind |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #487. Ports the 7 complex CELT float kernels — the IMDCT/MDCT
rotation, twiddle and fold kernels — from hand Plan9 asm to Go
simd/archsimd,gated
goexperiment.simd && !purego:imdct_pre_kiss,imdct_tdac,imdct_post_kiss,mdct_post_twiddle,mdct_mid_fold,mdct_fold1,mdct_fold3.These were the last CELT float kernels blocked on a missing op: a full 4-lane
lane-reverse. They use
ConcatEven/ConcatOddto deinterleave thekissCpxre/im streams, a
reverse4(built from the newReverse64/NEONVREV64) for thedescending streams, fused
MulAdd+ single-roundMulmatching each kernel'sexact op shape, and
InterleaveLo/Hifor the complex store. The bit-reverseddst[bitrev[i]]scatter (fold kernels) is done via temps + a scalar loop, exactlyas the asm scatters it.
Honest state — needs the Reverse64 op (not upstream yet)
reverse4needsFloat32x4.Reverse64, which is not in upstream Go yet —submitted as golang/go#80032. Until it lands in
gotip:.s/.gofiles retag
!goexperiment.simd);goexperiment.simdbuild of these kernels needs the patched toolchain(
thesyncim/go), so the non-blockingtest-simd-experimentCI job (upstreamgotip) will be red on the arm64 cross-build until #80032 merges — this is
expected and the PR assumes that merge.
Locally on the patched gotip the full set (vanilla + widen + complex) builds and
every kernel's parity test passes on both the archsimd and asm paths.
What we win
archsimd matches the hand asm at parity on the complex family — the
deinterleave + reverse + scatter amortize to parity, not the regression you'd
expect from emulated VLD2:
imdct_post_kissimdct_tdacimdct_pre_kissmdct_post_twiddle/mdct_mid_fold/mdct_fold1/3With these merged, every CELT float kernel is portable to archsimd; the only
remaining hand asm is the genuinely-blocked classes (FFT butterfly gather,
indexed-lane FMLA, serial IIR, integer).
Test plan
GOEXPERIMENT=simdpatched gotip AND normalgo test(asm) AND-tags puregointernal/celtsuite green on both pathstest-simd-experimentCI: red until simd: add ARM64 Float32x4.Reverse64 (REV64) golang/go#80032 lands in upstream gotip (expected)