Skip to content

fix: prevent UTF-8 corruption at chunk read boundaries (#521) - #524

Open
A-S-Manoj wants to merge 1 commit into
VoidenHQ:betafrom
A-S-Manoj:fix/utf8-chunk-boundary-corruption
Open

fix: prevent UTF-8 corruption at chunk read boundaries (#521)#524
A-S-Manoj wants to merge 1 commit into
VoidenHQ:betafrom
A-S-Manoj:fix/utf8-chunk-boundary-corruption

Conversation

@A-S-Manoj

Copy link
Copy Markdown

Summary

Fixes #521 — file content read via readChunk could corrupt multi-byte UTF-8 characters (e.g. é) when a fixed-size chunk boundary happened to fall in the middle of their byte sequence.

Root cause

Each chunk was decoded independently with buf.toString('utf8'). When a multi-byte character's bytes were split across two chunks, each half was decoded on its own, producing replacement characters () instead of the original character.

Fix

  • Added trailingIncompleteUtf8Length(buf), which detects whether a buffer ends mid-way through a multi-byte UTF-8 sequence and returns how many trailing bytes to hold back.
  • readChunk now trims any incomplete trailing sequence from the current chunk (unless it's the final chunk) and only advances nextOffset past the bytes actually decoded. The held-back bytes are naturally picked up as the start of the next chunk.
  • Guarded against pathologically small chunk sizes ever holding back an entire chunk (which would stall nextOffset).

Testing

  • Unit tests for trailingIncompleteUtf8Length covering 1/2/3/4-byte UTF-8 sequences cut at every possible split point, plus an exhaustive round-trip test across all split points of a mixed-width string.
  • Regression test reproducing the exact scenario from Preserve multibyte code points across fixed byte-chunk boundaries #521: a two-byte character (é) straddling the 512 KiB chunk boundary.
  • Multi-chunk-size test (64B–512KiB) verifying full-file content is preserved regardless of chunk size.

@phurpa-tsering
phurpa-tsering changed the base branch from main to beta July 29, 2026 11:20
@A-S-Manoj
A-S-Manoj force-pushed the fix/utf8-chunk-boundary-corruption branch from 7b80a95 to 3e55d54 Compare July 29, 2026 12:17
@phurpa-tsering

Copy link
Copy Markdown
Collaborator

hey @A-S-Manoj
Nice fix — the trailingIncompleteUtf8Length logic and test coverage look solid.

This branch picked up some unrelated commits during a rebase onto beta (the choco/winget/dependencybot changes — 15 files instead of the 2 that actually fix the bug). Could you rebase onto the current beta ?

@A-S-Manoj
A-S-Manoj force-pushed the fix/utf8-chunk-boundary-corruption branch from 3e55d54 to 48f14f4 Compare July 30, 2026 18:35
@A-S-Manoj

Copy link
Copy Markdown
Author

@phurpa-tsering Done, rebased onto current beta. Down to the one fix commit, 2 files now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve multibyte code points across fixed byte-chunk boundaries

2 participants