Add "Remove/replace Unicode characters" tool (SE 4 plugin port)#12596
Open
niksedk wants to merge 1 commit into
Open
Add "Remove/replace Unicode characters" tool (SE 4 plugin port)#12596niksedk wants to merge 1 commit into
niksedk wants to merge 1 commit into
Conversation
SE 5 native port of the SE 4 "Remove Unicode characters" plugin (issue #12586): a Tools-menu window that finds every character above Latin-1 (code point > 255) in the subtitle and lists each with its code point, occurrence count and line numbers. Checked characters are removed, or replaced via an editable per-character "replace with" column. Non-empty mappings are remembered in Settings.json (Tools.RemoveUnicodeCharacters.ReplaceList, seeded with the plugin's defaults: musical notes to #). Unlike the SE 4 plugin, characters are scanned per rune, so surrogate pairs (emoji etc.) are treated as one character instead of two broken halves. Deliberately a standalone interactive tool rather than a Fix Common Errors rule: for non-Latin subtitles (Russian, Greek, Arabic, CJK) every character is above 255, so anything non-interactive built on this detection would offer to destroy the whole file. Fixes #12586 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12586 — native SE 5 port of the SE 4 "Remove Unicode characters" plugin.
What it does: Tools → Remove/replace Unicode characters... scans the subtitle for every character above Latin-1 (code point > 255) and lists each one with its code point (
U+266Astyle), occurrence count, and the line numbers it appears in. Each row has an Apply checkbox and an editable Replace with cell — empty means remove. Select all / invert selection buttons, status line with the total count. On OK the changes are applied to the subtitle via the sameApplyFixedSubtitleflow as Convert Actors.Replace list persistence: non-empty "replace with" mappings are remembered in
Settings.json(Tools.RemoveUnicodeCharacters.ReplaceList) and prefilled the next time the same character shows up — seeded with the SE 4 plugin's defaults (♪/♫→#). Clearing a mapping removes it from the list.One improvement over the SE 4 plugin: scanning is per rune, not per
char— the SE 4 plugin iterated UTF-16 code units, so an emoji showed up as two broken surrogate halves and replacing one corrupted the text. Here👍is listed (and removed/replaced) as one character.Why a standalone tool and not a Fix Common Errors rule (per the issue discussion): the tool's detection principle is ">255 is suspect", and for Russian, Greek, Arabic, or CJK subtitles every character is above 255 — anything non-interactive would offer to destroy the whole file. The interactive checkable list keeps it the user's informed choice, same as the plugin. The curated-mapping side of this space is already covered by the "Normalize strings" FCE rule.
Wiring: feature folder under
Features/Tools/RemoveUnicodeCharacters/(window + view model + display item, following the Convert Actors pattern), DI registration, Tools menu entry (alphabetically sorted group), language strings underTools.RemoveUnicodeCharacters, settings class.Tests: four view-model tests — scan groups characters and ignores Latin-1 (é), surrogate pair listed as one character with
U+01F44D, replace/remove/unchecked-keeps behavior through OK, empty-result status, and select-all/invert. All 695 UI tests pass; full solution builds clean.Not merging per request — the window layout is probably worth a visual once-over (built after the Convert Actors window style: 800×600, resizable, DataGrid with checkbox multi-select support).
🤖 Generated with Claude Code