libretro-common: streaming Ogg for rvorbis, windowed reads for rwav, rflac push-source fixes - #19316
Merged
Conversation
added 5 commits
July 31, 2026 20:35
rwav_decode_s16 kept a 256KB static scratch for windows starting or ending inside a coded block: the block was decoded whole into it and the wanted part copied out. That made the function non-reentrant, and the comment acknowledged it on the reasoning that the whole-file decode audio_mixer performs lands on the block-aligned fast path every time. A windowed caller inverts that. Reading in fixed-size chunks is mid-block on nearly every call, so the scratch would be in constant use and two voices decoding ADPCM concurrently would corrupt each other. Give rwav_decode_block the window instead. A block must still be decoded from its start - the predictor state is what its samples continue from - but nothing before the window has to be stored, so the mid-block case writes straight into the caller's buffer and the scratch is not needed rather than merely made safe. Both call sites collapse into one. TSan, eight threads decoding overlapping mid-block windows of one IMA file: data races at rwav.c:570 and :602 before, none after. Output verified identical to the previous code across 308 combinations - 11 fixtures covering u8, s16, s24, f32, A-law, mu-law, MS ADPCM and IMA ADPCM in stereo and mono, seven window steps down to a single frame, and four starting offsets including mid-block.
rwav_decode_s16 addresses frames, but a caller streaming a file has bytes, and nothing said which ones a frame range required. For the uncompressed and companded formats that is blockalign arithmetic the caller could do itself. For the block-coded ones it is not: a frame lives inside a coded block only rwav can locate, and fetching a narrower span decodes to noise rather than failing, because a block read from its middle still yields samples. rwav_frame_extent reports the byte range, widened to whole blocks where that matters. rwav_decode_s16_at decodes from a chunk holding just that range, checking rather than trusting that it starts where the extent said - a caller whose arithmetic is wrong should find out at once instead of hearing it. Nothing existing changes; rwav_decode_s16 is untouched and its callers need no adjustment. Verified against the whole-buffer decode across 11 fixtures and six window sizes, with each chunk copied into an allocation of exactly the reported extent so that any read past it is a heap overflow under ASan. Clean, which is what shows the extent is sufficient and not merely plausible.
The pulling API addresses the whole stream at once - it seeks by bisecting over a resident buffer - so a caller must hold the file to use it at all. A caller reading from storage has no such buffer, and fabricating one costs the whole file in memory to play a few kilobytes at a time. Nothing here decodes. rvorbis already had a raw-packet decoder built for Matroska, which reads a packet where it lies; what was missing was an Ogg layer that could be walked incrementally. So this is a demuxer - page headers, the lacing table, packet reassembly across page boundaries, serial filtering, byte-at-a-time resync - handing packets to the decoder that was already there. Residency is one packet rather than one file. The stb-derived decode body is untouched. Shaped like rflac: set_in, set_out, process. Input is consumed rather than borrowed, so a window may slide freely. Three things the format forces. A stream's last packet decodes past the end of the audio, and where that end is is stated only by the final page's granule - known before that page's packets are decoded, so the overhang is dropped rather than emitted and retracted. A resync lands inside a page, where nothing says which frames the output belongs to; rvorbis_stream_tell reports the position from the first page boundary crossed, so a caller need not reimplement the arithmetic to seek exactly. And the first packets after a resync reference an overlap that was never decoded, so failing to decode them is how converging looks rather than a broken stream - they are skipped until one succeeds. Verified against the resident decoder as oracle: bit-exact on three synthetic fixtures across 120 combinations of window and output size, down to one-byte input windows and single-frame outputs, and on real CD audio tracks; eight seek targets bit-exact at the sought frame; some 3,400 damaged streams covering every truncation length, garbage prefixes and random corruption, with no crash, hang or leak. ASan, UBSan and LeakSan clean, TSan clean with six concurrent instances. Chained and multiplexed Ogg are untested - the demuxer locks onto the first Vorbis bitstream, so multiplexed should work and chained will stop at the first link, but neither is covered by the fixtures.
A caller feeding rflac a sliding window decoded one correct block and
then quietly wrong ones, ending some 38% into the stream with no error
reported.
*read reported bytes whose bits had been consumed, which lags what the
bitreader pulled from the source by whatever sits in its cache. That
cache survives across calls, so reporting the smaller figure tells the
caller to present those bytes again and the bitreader reads them twice -
once from the cache, once from the new span - and the stream runs ahead
of itself by the cache depth every call. It stays plausible rather than
failing because a FLAC frame header is a sync pattern the decoder
resynchronises on.
But that figure is not simply wrong: rchd's CD FLAC codec uses it to
tell a hunk's audio from the subchannel packed after it, which needs
logical consumption precisely because it lags. They are two quantities.
*read keeps its meaning; rflac_span_taken reports the other, and a
windowed caller advances by that. It is cleared on entry, not only on
the paths that set it, so a call returning at a guard cannot leave a
figure behind for the caller to advance by twice.
Two more, both found by the same caller:
- A stream's last frame is shorter than the longest one it could
legally be, so the tail of every file landed in the rollback path
and waited for bytes that were not coming. rflac_set_eof states
that nothing more is arriving.
- An underrun on a span longer than any legal frame was read as end of
stream. It is not - the bitreader reads ahead of the frame it is
decoding, so it underruns on spans that comfortably hold the frame,
and the stream was silently latched as ended. What ends a stream is
now only the caller saying so.
rflac_min_input reports a window size that always makes progress. It is
deliberately generous: below roughly 16k, decoding degrades on streams
whose frames are far smaller than that, for reasons the format does not
explain, so it is documented as a measured floor rather than a bound.
Verified bit-exact against ffmpeg on five fixtures spanning compression
levels 0, 5, 8 and default, on real CD audio, across window sizes from
16k to 128k, and through two real NeoGeo CD images plus six synthetic
ones covering cdzl, cdzs, cdlz and cdfl - all decoding identically.
audio_transfer's FLAC arm was exact before and remains so; it feeds
whole buffers and never met the bug.
…t whole The self-framed .ogg path opened the entire buffer with rvorbis_open_memory and seeked by bisecting over it. rvorbis now walks Ogg incrementally, so the arm feeds it instead and holds the decoder it returns; only the setup headers need be resident to open. Seeking becomes an explicit bisection on page granule with a bounded forward decode, and buffer_tell reports the demuxer's own cursor. Worth being plain about what this does not do. set_buffer_ptr still takes one base pointer for the whole file, so a caller still allocates the whole file whatever the decoder does underneath, and the footprint does not move. This is groundwork - it makes rvorbis_stream the single Ogg path and gives the arm a better-behaved seek - not the memory win itself, which needs the buffer interface to change. The multiplexed, WebM and demuxed arms are untouched: they filter by serial or take packets directly, and the resident page walker still serves the Opus arm. Verified against the unmodified file as oracle: byte-identical across 12 combinations of chunk size and 10 seek targets, ASan, UBSan and LeakSan clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five commits against
libretro-common, adding windowed/streaming decode torwavandrvorbis, fixing three real bugs inrflac's push-source layer and one reentrancy bug inrwav, and movingaudio_transfer's plain-Ogg arm onto the new rvorbis API.The motivation is a companion series in
neocd_libretrothat dropslibchdr,libogg,libvorbisandminizipin favour ofrchd/rvorbis/rflac/rwav, but the tworwavcommits and the
rflaccommit are standalone bug fixes worth taking regardless.7 files changed, 1258 insertions(+), 68 deletions(-)Bugs fixed
rwavwas not reentrantrwav_decode_s16kept a 256KBstaticscratch buffer for windows starting or ending insidea coded ADPCM block. The existing comment acknowledged this, reasoning that
audio_mixer'swhole-file decode lands on the block-aligned fast path every time. A windowed caller inverts
that — fixed-size chunks are mid-block on nearly every call.
TSan, eight threads decoding overlapping mid-block windows of one IMA file:
The fix gives
rwav_decode_blockan emission window rather than making the scratch safe. Ablock must still be decoded from its start — the predictor state is what its samples
continue from — but nothing before the window has to be stored, so the mid-block case
writes straight into the caller's buffer and both call sites collapse into one.
rflacsilently corrupted windowed decodesA caller feeding
rflaca sliding window decoded one correct block and then quietly wrongones, ending ~38% into the stream with no error reported.
*readreported bytes whose bits had been consumed, which lags what the bitreader pulledfrom the source by whatever sits in its cache. That cache survives across calls, so reporting
the smaller figure tells the caller to present those bytes again and the bitreader reads them
twice — once from cache, once from the new span. The stream runs ahead of itself by the cache
depth every call. It stays plausible rather than failing because a FLAC frame header is a sync
pattern the decoder resynchronises on.
That figure is not simply wrong, though:
rchd's CD FLAC codec uses it to tell a hunk's audiofrom the subchannel packed after it, which needs logical consumption precisely because it
lags. They are two quantities.
*readkeeps its meaning;rflac_span_taken()reports theother.
Two more, both surfaced by the same caller:
longest one it could legally be, so it landed in the rollback path and waited for bytes that
were never coming.
rflac_set_eof()states that nothing more is arriving.ahead of the frame it's decoding, so it underruns on spans that comfortably hold the frame,
and the stream was silently latched as ended.
New API
The rvorbis addition decodes nothing. rvorbis already had a raw-packet decoder built for
Matroska which reads a packet where it lies; what was missing was an Ogg layer that could be
walked incrementally. The new code is purely a demuxer — page headers, lacing table, packet
reassembly across page boundaries, serial filtering, byte-at-a-time resync — feeding the
decoder that was already there. The stb-derived decode body is untouched, which is why the
line count is larger than the risk.
Verification
Every change was checked differentially against pristine code compiled from the untouched
tree, or against
ffmpegwhere there was no in-tree oracle.rwavscratch removalrwavrange APIrvorbisrflacaudio_transferASan + UBSan + LeakSan clean throughout; TSan clean with six concurrent rvorbis instances and
eight concurrent rwav decodes.
Also exercised end-to-end through two real NeoGeo CD images (
cdlz/cdzl/cdfl,CHT2metadata), comparing every track against
chdman's own extraction — 485,713 sectors, nomismatch — and against real CD audio converted to WAV/FLAC/OGG. Notably,
rvorbisandlibvorbisagree to mean 0.00, max 1 LSB on that material.What this does not do
audio_transferchange buys no memory on its own.set_buffer_ptrstill takes onebase pointer for the whole file, so a caller allocates the whole file whatever the decoder
does underneath. I tested this rather than assumed it — poisoning every byte behind the
reported frontier leaves output unchanged both before and after. It's groundwork plus a
better-behaved seek; the actual win needs the buffer interface to change, which is a separate
design question (a pull-source callback is the obvious shape but would run I/O on the audio
thread under a voice lock, so probably not that).
rflac_min_input()is measured, not derived. Below roughly 16k, decoding degrades onstreams whose frames are far smaller than that, for reasons the format doesn't explain. It's
documented as a conservative floor rather than a tight bound.
bitstream, so multiplexed should work and chained will stop at the first link, but neither is
covered by the fixtures.
audio_transferarms are untouched, and the resident pagewalker still serves the Opus arm.