Rebuild ANSI measurement, truncation, and wrapping on one sequence grammar - #624
Open
GoodForOneFare wants to merge 7 commits into
Open
Rebuild ANSI measurement, truncation, and wrapping on one sequence grammar#624GoodForOneFare wants to merge 7 commits into
GoodForOneFare wants to merge 7 commits into
Conversation
CSI_SEQUENCE required at least one parameter character, so strip_codes
passed \e[K, \e[m, and private-mode sequences like \e[?25l through as
text, and printing_width counted their bytes as printed columns:
printing_width("\e[?25lx\e[K") returned 10 for one visible character.
Match the full CSI grammar instead: any parameter bytes (including the
private-mode markers), then intermediate bytes, then one final byte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GoodForOneFare
marked this pull request as ready for review
August 11, 2026 18:53
ANSI.each_token walks a string as alternating runs of whole control sequences (CSI or OSC) and the text between them, so consumers that measure or cut at token boundaries can no longer slice a sequence open or count its bytes as printable. printing_width is rebuilt on top of it, fixing three bugs: - Newlines counted as one column (the old `when "\n"` branch compared a String against Integer codepoints, so it never matched); they now count as zero. - Emoji counted as one column while Truncater counted them as two; both now share ANSI.grapheme_width, which says two. - ZWJ sequences and combining marks were hand-rolled or miscounted; String#grapheme_clusters now groups them, so 👩💻 and e+U+0301 are each one cluster. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Truncater's hand-rolled codepoint state machine predated the fixes to CSI_SEQUENCE and knew even less of the grammar: its parameter-byte set had no `?`, so a private-mode sequence like \x1b[?25l was sliced after the `?` and its body counted as printable text, and OSC sequences weren't recognized at all, so truncating inside an OSC 8 hyperlink emitted a dangling, unterminated link. It now walks ANSI.each_token: sequences pass through whole and spend no width, text is measured and cut at grapheme-cluster boundaries (the ZWJ special-casing goes away with it), and if the cut lands inside an OSC 8 hyperlink the truncation suffix closes it before resetting SGR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wrap built its own lexer: an alternation of twenty lookbehind patterns (one per SGR parameter length, since lookbehinds can't quantify) spliced into a split regex. Walking ANSI.each_token replaces all of it, and any non-SGR sequence (cursor movement, OSC) now passes through as a unit instead of being split as text. It also fixes a dead branch: the reset case was written as the single-quoted literal '\x1B[0?m', which no token ever equals, so the SGR codes tracked for re-sending after each wrap were never cleared — a reset color could come back on the next wrapped line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CLI::UI.link built hyperlinks while Truncater parsed and closed them, each with its own copy of the escape-sequence grammar — the same split that motivated ANSI.each_token. Both halves now live in ANSI next to the other sequence definitions: HYPERLINK classifies a sequence and captures its URI, HYPERLINK_END closes a link, and ANSI.hyperlink builds one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The early return compared codepoint count against printing width, the
one measurement in the file grapheme_width doesn't make. Character
count only bounds column count for ASCII — an emoji string occupies up
to twice as many columns as it has characters, so
Truncater.call("🌈🌈🌈", 3) returned all six columns untouched. The
fast path now applies only to ASCII strings.
Also adopts ANSI's OSC 8 grammar in place of the local copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wrap resends the SGR codes in effect after each line break so frame coloring doesn't clobber them, but an OSC 8 hyperlink spanning a break was left open across the newline, putting the next line's frame gutter inside the link. Breaks now close an open hyperlink and reopen it after the resent codes, exactly as Truncater closes one at a cut. The SGR tracking also accepts the colon form of extended colors (\x1b[38:2::255:0:0m), which the previous [\d;] parameter set silently dropped from the resend list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GoodForOneFare
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This started as a one-regex fix and grew into consolidating every place that parsed ANSI by hand onto a single grammar. It changes measured widths for every consumer, so it warrants a minor version bump, not a patch — details under "Behavior changes" below.
The grammar
CSI_SEQUENCErequired at least one parameter character, sostrip_codespassed\e[K,\e[m, and private-mode sequences like\e[?25lthrough as text, andprinting_widthcounted their bytes as printed columns:printing_width("\e[?25lx\e[K")returned10for one visible character. It now matches the full CSI grammar (any parameter bytes including private-mode markers, then intermediate bytes, then one final byte). The OSC 8 hyperlink grammar also moves intoANSI:CLI::UI.linkbuilds links andTruncater/Wrapclose them from the same definitions.The walk
ANSI.each_tokenscans a string as alternating runs of whole control sequences and the text between them, so consumers that measure or cut at token boundaries can't slice a sequence open or count its bytes as printable.printing_width,Truncater, andWrapare all rebuilt on it, replacing three independent hand-rolled parsers (one of which, Truncater's, didn't know about?parameters or OSC at all — it sliced\e[?25lin half and could emit a dangling unterminated hyperlink).Behavior changes
printing_width: emoji are 2 columns (was 1; Truncater always said 2 — they now agree via sharedANSI.grapheme_width), newlines and combining marks are 0 columns (was 1), and ZWJ sequences are measured as single grapheme clusters. Table columns, frame padding, and spin_group truncation all shift for emoji-containing content; downstream repos with exact-output assertions will churn.Truncater.call: no longer slices private-mode sequences mid-way or counts their bodies as text; closes an open OSC 8 hyperlink at the cut; measures by columns even on the fast path (call("🌈🌈🌈", 3)previously returned all six columns untouched because the early-out compared character count).Wrap: an\e[0mreset now actually clears the SGR codes resent after each break — the old reset branch was the single-quoted literal'\x1B[0?m', which no token ever equals, so codes accumulated forever and a reset color could come back on the next wrapped line. Colon-form extended colors (\e[38:2::255:0:0m) are now tracked too, and a hyperlink spanning a break is closed and reopened so the frame gutter stays outside the link.🤖 Generated with Claude Code