fix: prevent UTF-8 corruption at chunk read boundaries (#521) - #524
Open
A-S-Manoj wants to merge 1 commit into
Open
fix: prevent UTF-8 corruption at chunk read boundaries (#521)#524A-S-Manoj wants to merge 1 commit into
A-S-Manoj wants to merge 1 commit into
Conversation
A-S-Manoj
force-pushed
the
fix/utf8-chunk-boundary-corruption
branch
from
July 29, 2026 12:17
7b80a95 to
3e55d54
Compare
Collaborator
|
hey @A-S-Manoj This branch picked up some unrelated commits during a rebase onto beta (the choco/winget/dependencybot changes — 15 files instead of the 2 that actually fix the bug). Could you rebase onto the current beta ? |
A-S-Manoj
force-pushed
the
fix/utf8-chunk-boundary-corruption
branch
from
July 30, 2026 18:35
3e55d54 to
48f14f4
Compare
Author
|
@phurpa-tsering Done, rebased onto current beta. Down to the one fix commit, 2 files now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #521 — file content read via
readChunkcould corrupt multi-byte UTF-8 characters (e.g.é) when a fixed-size chunk boundary happened to fall in the middle of their byte sequence.Root cause
Each chunk was decoded independently with
buf.toString('utf8'). When a multi-byte character's bytes were split across two chunks, each half was decoded on its own, producing replacement characters (�) instead of the original character.Fix
trailingIncompleteUtf8Length(buf), which detects whether a buffer ends mid-way through a multi-byte UTF-8 sequence and returns how many trailing bytes to hold back.readChunknow trims any incomplete trailing sequence from the current chunk (unless it's the final chunk) and only advancesnextOffsetpast the bytes actually decoded. The held-back bytes are naturally picked up as the start of the next chunk.nextOffset).Testing
trailingIncompleteUtf8Lengthcovering 1/2/3/4-byte UTF-8 sequences cut at every possible split point, plus an exhaustive round-trip test across all split points of a mixed-width string.é) straddling the 512 KiB chunk boundary.