Add complex-scripts feature for CJK and Southeast Asian line/word breaking - #621
Merged
Merged
Conversation
…word breaking
When enabled, uses ICU4X dictionary-based segmenters (LineSegmenter::new_dictionary,
WordSegmenter::new_dictionary) that include CJK (Chinese, Japanese) and Southeast Asian
(Thai, Khmer, Lao, Myanmar) dictionaries for proper line and word breaking.
When disabled (default), the lightweight non-complex segmenter is used, which falls back
to character-level breaks for those scripts.
Usage:
parley = { features = ["complex-scripts"] }
Linktime
changed the title
complex-scripts feature for CJK and Southeast Asian line/word breaking
nicoburns
approved these changes
May 24, 2026
nicoburns
left a comment
nicoburns
left a comment
Collaborator
There was a problem hiding this comment.
Thanks! This looks great. Sorry for the delayed review: most of the Parley developers were at RustWeek last week.
Comment on lines
58
to
74
| match word_break_strength { | ||
| WordBreak::Normal => { | ||
| const { | ||
| let mut opt = LineBreakOptions::default(); | ||
| opt.word_option = Some(LineBreakWordOption::Normal); | ||
| LineSegmenter::new_for_non_complex_scripts(opt) | ||
| } | ||
| let mut opt = LineBreakOptions::default(); | ||
| opt.word_option = Some(LineBreakWordOption::Normal); | ||
| line_segmenter_impl(opt) | ||
| } | ||
| WordBreak::BreakAll => { | ||
| const { | ||
| let mut opt = LineBreakOptions::default(); | ||
| opt.word_option = Some(LineBreakWordOption::BreakAll); | ||
| LineSegmenter::new_for_non_complex_scripts(opt) | ||
| } | ||
| let mut opt = LineBreakOptions::default(); | ||
| opt.word_option = Some(LineBreakWordOption::BreakAll); | ||
| line_segmenter_impl(opt) | ||
| } | ||
| WordBreak::KeepAll => { | ||
| const { | ||
| let mut opt = LineBreakOptions::default(); | ||
| opt.word_option = Some(LineBreakWordOption::KeepAll); | ||
| LineSegmenter::new_for_non_complex_scripts(opt) | ||
| } | ||
| let mut opt = LineBreakOptions::default(); | ||
| opt.word_option = Some(LineBreakWordOption::KeepAll); | ||
| line_segmenter_impl(opt) | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
This could probably be simpliflied to:
#[inline(always)]
fn line_segmenter(&self, word_break_strength: WordBreak) -> LineSegmenterBorrowed<'static> {
let mut opt = LineBreakOptions::default();
opt.word_option = Some(match word_break_strength {
WordBreak::Normal => LineBreakWordOption::Normal,
WordBreak::BreakAll => LineBreakWordOption::BreakAll,
WordBreak::KeepAll => LineBreakWordOption::KeepAll,
});
line_segmenter_impl(opt)
}But I'm not going to require it as part of this PR seeing it was a pre-existing issue.
Contributor
|
There was an existing PR for this that would allow adding models more granularly, and at runtime: #533 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
complex-scriptsCargo feature that switches the line and word segmenters from the lightweightnew_for_non_complex_scripts()tonew_dictionary(), enabling proper dictionary-based line and word breaking for complex scripts:cjdictdictionary for word-boundary detectionWhen the feature is disabled (default), behavior is unchanged: the lightweight segmenter is used, which falls back to character-level breaks for the above scripts.
Motivation
Without this feature, rendering CJK text produces a runtime warning from ICU4X:
This occurs because
new_for_non_complex_scripts()does not include thecjdictdictionary data. CJK text still renders, but line breaks are character-level rather than word-level.Usage
Changes
parley/Cargo.toml: addcomplex-scriptsfeature flagparley/src/analysis/mod.rs: cfg-gateline_segmenterandword_segmenterto useLineSegmenter::new_dictionary()/WordSegmenter::new_dictionary()when enabledCHANGELOG.md: document new featureTesting
Both configurations compile cleanly:
cargo check -p parley— default (no feature)cargo check -p parley --features complex-scripts— with feature enabled