Replace printables table with `unicode_data.rs` tables by Jules-Bertholet · Pull Request #155527 · rust-lang/rust

Jules-Bertholet · 2026-04-19T21:46:54Z

This gets rid of the printable.py script, ensuring that unicode-table-generator handles all our Unicode data table generation needs.

There are also some drive-by documentation improvements in library/core/char/methods.rs.

There is one change in behavior: we now consider all characters with the Default_Ignorable_Code_Point property to be unprintable. These characters can be hidden/invisible otherwise.

I've chosen to give each Unicode property its own table, instead of merging them all into one. This is slightly less efficient in terms of space, but should allow us to expose these tables in the future with public methods on char.

@rustbot label A-Unicode

rustbot · 2026-04-19T21:46:58Z

library/core/src/unicode/unicode_data.rs is generated by the src/tools/unicode-table-generator tool.

If you want to modify unicode_data.rs, please modify the tool then regenerate the library source file via ./x run src/tools/unicode-table-generator instead of editing unicode_data.rs manually.

rustbot · 2026-04-19T21:47:00Z

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: @scottmcm, libs
@scottmcm, libs expanded to 7 candidates
Random selection from Mark-Simulacrum, jhpratt, scottmcm

Mark-Simulacrum · 2026-04-26T18:49:45Z

There is one change in behavior: we now consider all characters with the Default_Ignorable_Code_Point property to be unprintable. These characters can be hidden/invisible otherwise.

Nominating for libs-api to FCP this. @Jules-Bertholet can you write up how that affects the public API of std? i.e., where is that unprintability used (only in Debug impls of str)?

Mark-Simulacrum

Can you split out the re-ordering and renaming in char/methods.rs? It's very hard to review the diff for me when methods are moved around in the file. It also seems entirely unrelated to the core change here and I'd rather have separate commits at least.

The changes look broadly reasonable though, I'd be happy to accept them if separated out (including maybe from the libs-api facing change).

View changes since this review

rustbot · 2026-04-26T18:57:28Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

Jules-Bertholet · 2026-04-26T22:56:02Z

@rustbot ready

6079a98 is the libs-API-relevant change. It affects the Debug impls for char and the various string types, as well as the escape_debug() methods on char and str. The following characters are changed to be always escaped: https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=%5Cp%7BDefault_Ignorable_Code_Point%7D-%5Cp%7BCf%7D-%5Cp%7BCn%7D

Note that we may also wish to stop escaping format control characters which are not default-ignorable. The list of characters this would affect: https://util.unicode.org/UnicodeJsps/list-unicodeset.jsp?a=%5Cp%7BCf%7D-%5Cp%7BDefault_Ignorable_Code_Point%7D

nia-e · 2026-04-28T17:34:59Z

We discussed this in today's @rust-lang/libs-api meeting; +1 for us, but we'd like someone with more unicode knowledge to weigh in to be safe so cc @Manishearth (offtopic, it would be nice to have an @rust-lang/unicode-knowers ping group since these issues arise pretty often)

Manishearth · 2026-04-28T20:28:54Z

I didn't look too closely, but this seems fine. From a quick look the printability concern is for debug output, and yes, being more conservative there makes sense.

Manishearth · 2026-04-28T20:29:52Z

While you shouldn't depend on ICU4X in the stdlib, it may be worth using ICU4X to get your unicode properties, instead of fetching them yourself. This does mean you are beholden to ICU4X for unicode updates though.

Amanieu · 2026-05-05T15:28:02Z

@rfcbot merge libs-api

rust-rfcbot · 2026-05-05T15:28:05Z

Team member @Amanieu has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

Mark-Simulacrum

r=me with FCP complete

View changes since this review

Rollup of 25 pull requests Successful merges: - rust-lang/rust#157251 (`rust-analyzer` subtree update) - rust-lang/rust#157533 (Subtree sync for rustc_codegen_cranelift) - rust-lang/rust#154742 (Add APIs for case folding to the standard library) - rust-lang/rust#155144 (mir_build: Add an extra intermediate step in MIR building for patterns ) - rust-lang/rust#156222 (Stabilize `Result::map_or_default` and `Option::map_or_default`) - rust-lang/rust#157016 (add `extern "tail"` calling convention) - rust-lang/rust#157264 (diagnostics: Fix ICE building a trait ref in method suggestions) - rust-lang/rust#157386 (Parse deprecated note links separately in rustc_resolve) - rust-lang/rust#157483 (fix windows-gnu TLS leak) - rust-lang/rust#157488 (compiletest: inject `#![windows_subsystem = "windows"]` to debuginfo tests on Windows) - rust-lang/rust#157509 (remove solaris implementation for File::lock, it has the wrong semantics) - rust-lang/rust#157521 (Rename `SyncView::{as_pin => as_pin_ref}`) - rust-lang/rust#156136 (Move tests box) - rust-lang/rust#156573 (Add unwinder_private_data_size for wasm64 target) - rust-lang/rust#156783 (docs: make `Rc::into_raw` clickable in `Rc::increment_strong_count` doc) - rust-lang/rust#156840 (Stabilize `PathBuf::into_string`) - rust-lang/rust#156936 (Remove FIXME about impl PinCoerceUnsized for UnsafePinned<T>) - rust-lang/rust#157365 (Revert "LLVM 23: Run AssignGUIDPass in some places") - rust-lang/rust#157380 (clarify compiler_fence (and fence) docs) - rust-lang/rust#157471 (Debug assert that parsed attributes are in the `BUILTIN_ATTRIBUTE_MAP`) - rust-lang/rust#157485 (Rename `errors.rs` file to `diagnostics.rs` (1/N)) - rust-lang/rust#157494 (Convert `QueryRegionConstraint` into a struct) - rust-lang/rust#157526 (std tests: skip a slow test on Miri) - rust-lang/rust#157531 (ci: bump x86_64-gnu base image to 26.04) - rust-lang/rust#157556 (Add `BTree::append()` change to 1.96.0 relnotes) Failed merges: - rust-lang/rust#155527 (Replace printables table with `unicode_data.rs` tables)

And rename a struct field.

This gets rid of the `printable.py` script, ensuring that `unicode-table-generator` handles all our Unicode data table generation needs. I've elected to give each Unicode property its own table, instead of merging them all into one. This is slightly less efficient in terms of space, but should allow us to expose these tables in the future with public methods on `char`.

These characters may be hidden/invisible otherwise.

rustbot · 2026-06-09T04:36:38Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

Jules-Bertholet · 2026-06-09T04:39:08Z

@rustbot ready

Mark-Simulacrum · 2026-06-11T01:54:53Z

@bors r+

rust-bors · 2026-06-11T01:54:56Z

📌 Commit 4db9ff5 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@rustbot

…Mark-Simulacrum Replace printables table with `unicode_data.rs` tables This gets rid of the `printable.py` script, ensuring that `unicode-table-generator` handles all our Unicode data table generation needs. There are also some drive-by documentation improvements in `library/core/char/methods.rs`. There is one change in behavior: we now consider all characters with the `Default_Ignorable_Code_Point` property to be unprintable. These characters can be hidden/invisible otherwise. I've chosen to give each Unicode property its own table, instead of merging them all into one. This is slightly less efficient in terms of space, but should allow us to expose these tables in the future with public methods on `char`. @rustbot label A-Unicode

@rustbot

…Mark-Simulacrum Replace printables table with `unicode_data.rs` tables This gets rid of the `printable.py` script, ensuring that `unicode-table-generator` handles all our Unicode data table generation needs. There are also some drive-by documentation improvements in `library/core/char/methods.rs`. There is one change in behavior: we now consider all characters with the `Default_Ignorable_Code_Point` property to be unprintable. These characters can be hidden/invisible otherwise. I've chosen to give each Unicode property its own table, instead of merging them all into one. This is slightly less efficient in terms of space, but should allow us to expose these tables in the future with public methods on `char`. @rustbot label A-Unicode

@rustbot

…Mark-Simulacrum Replace printables table with `unicode_data.rs` tables This gets rid of the `printable.py` script, ensuring that `unicode-table-generator` handles all our Unicode data table generation needs. There are also some drive-by documentation improvements in `library/core/char/methods.rs`. There is one change in behavior: we now consider all characters with the `Default_Ignorable_Code_Point` property to be unprintable. These characters can be hidden/invisible otherwise. I've chosen to give each Unicode property its own table, instead of merging them all into one. This is slightly less efficient in terms of space, but should allow us to expose these tables in the future with public methods on `char`. @rustbot label A-Unicode

@rustbot

…Mark-Simulacrum Replace printables table with `unicode_data.rs` tables This gets rid of the `printable.py` script, ensuring that `unicode-table-generator` handles all our Unicode data table generation needs. There are also some drive-by documentation improvements in `library/core/char/methods.rs`. There is one change in behavior: we now consider all characters with the `Default_Ignorable_Code_Point` property to be unprintable. These characters can be hidden/invisible otherwise. I've chosen to give each Unicode property its own table, instead of merging them all into one. This is slightly less efficient in terms of space, but should allow us to expose these tables in the future with public methods on `char`. @rustbot label A-Unicode

Rollup of 31 pull requests Successful merges: - #141030 (Expand free alias types during variance computation) - #154853 (mgca: Register `ConstArgHasType` when normalizing projection consts) - #155527 (Replace printables table with `unicode_data.rs` tables) - #156629 (Stabilize `core::range::{legacy, RangeFull, RangeTo}`) - #157280 (traits: Allow escaping self types in ExistentialTraitRef::with_self_ty) - #157282 (Fix post-monomorphization error note race in the parallel frontend) - #157352 (Make the retained dep graph deterministic under the parallel frontend) - #157601 (Emit error for unused target expression in glob and list delegations) - #157611 (Update `browser-ui-test` version to `0.24.0`) - #157620 (Add a strategy FnMut to inject behavior into the flush cycle) - #157645 (Windows TLS - Only register the `atexit` hook when `cleanup` can be unloaded) - #157646 (Keep rename-imported main alive in dead-code analysis under `--test`) - #157647 (Start using comptime for reflection intrinsics and their wrapper functions) - #157719 (resolve: Partially revert "Remove a special case for dummy imports") - #155153 (Ensure Send/Sync is not implemented for std::env::Vars{,Os}) - #155198 (fix(mgca): Allow specifying generic args (of enum) on enum itself in unit & tuple variant constructions in (direct) const args) - #155323 (refactor `TypeRelativePath::AssocItem` to use `AliasTerm`) - #156497 (fix-155516: Don't suggest wrong unwrap expect) - #156583 (Support defaults for static EIIs) - #157013 (Ensure inferred let pattern types are well-formed) - #157196 (Only suggest reborrowing a moved value where the reborrow is valid) - #157230 (borrowck: avoid ICE describing fields on generic params) - #157288 (platform support: add SNaN erratum to MIPS targets) - #157330 (remove `IsTypeConst` from `hir::TraitItemKind`) - #157350 (compiletest: ignore SVG `y` offset in by-lines comparison) - #157355 (Add `or_try_*` variants for `HashMap` and `BTreeMap` Entry APIs) - #157577 (Fix parser error recovery treating 'dyn' as a strict keyword in Rust 2015 when used in `dyn + dyn`) - #157670 (Rename `errors.rs` file to `diagnostics.rs` (4/N)) - #157691 (Move symbol hiding code to a new file) - #157700 (Rename `errors.rs` file to `diagnostics.rs` (5/N)) - #157703 (Fix doc link to Instant sub in saturating caveat) Failed merges: - #157699 (Arg splat experiment - hir FnDecl impl)

@rustbot

Rollup merge of #155527 - Jules-Bertholet:riir-printable, r=Mark-Simulacrum Replace printables table with `unicode_data.rs` tables This gets rid of the `printable.py` script, ensuring that `unicode-table-generator` handles all our Unicode data table generation needs. There are also some drive-by documentation improvements in `library/core/char/methods.rs`. There is one change in behavior: we now consider all characters with the `Default_Ignorable_Code_Point` property to be unprintable. These characters can be hidden/invisible otherwise. I've chosen to give each Unicode property its own table, instead of merging them all into one. This is slightly less efficient in terms of space, but should allow us to expose these tables in the future with public methods on `char`. @rustbot label A-Unicode

Rollup of 31 pull requests Successful merges: - rust-lang#141030 (Expand free alias types during variance computation) - rust-lang#154853 (mgca: Register `ConstArgHasType` when normalizing projection consts) - rust-lang#155527 (Replace printables table with `unicode_data.rs` tables) - rust-lang#156629 (Stabilize `core::range::{legacy, RangeFull, RangeTo}`) - rust-lang#157280 (traits: Allow escaping self types in ExistentialTraitRef::with_self_ty) - rust-lang#157282 (Fix post-monomorphization error note race in the parallel frontend) - rust-lang#157352 (Make the retained dep graph deterministic under the parallel frontend) - rust-lang#157601 (Emit error for unused target expression in glob and list delegations) - rust-lang#157611 (Update `browser-ui-test` version to `0.24.0`) - rust-lang#157620 (Add a strategy FnMut to inject behavior into the flush cycle) - rust-lang#157645 (Windows TLS - Only register the `atexit` hook when `cleanup` can be unloaded) - rust-lang#157646 (Keep rename-imported main alive in dead-code analysis under `--test`) - rust-lang#157647 (Start using comptime for reflection intrinsics and their wrapper functions) - rust-lang#157719 (resolve: Partially revert "Remove a special case for dummy imports") - rust-lang#155153 (Ensure Send/Sync is not implemented for std::env::Vars{,Os}) - rust-lang#155198 (fix(mgca): Allow specifying generic args (of enum) on enum itself in unit & tuple variant constructions in (direct) const args) - rust-lang#155323 (refactor `TypeRelativePath::AssocItem` to use `AliasTerm`) - rust-lang#156497 (fix-155516: Don't suggest wrong unwrap expect) - rust-lang#156583 (Support defaults for static EIIs) - rust-lang#157013 (Ensure inferred let pattern types are well-formed) - rust-lang#157196 (Only suggest reborrowing a moved value where the reborrow is valid) - rust-lang#157230 (borrowck: avoid ICE describing fields on generic params) - rust-lang#157288 (platform support: add SNaN erratum to MIPS targets) - rust-lang#157330 (remove `IsTypeConst` from `hir::TraitItemKind`) - rust-lang#157350 (compiletest: ignore SVG `y` offset in by-lines comparison) - rust-lang#157355 (Add `or_try_*` variants for `HashMap` and `BTreeMap` Entry APIs) - rust-lang#157577 (Fix parser error recovery treating 'dyn' as a strict keyword in Rust 2015 when used in `dyn + dyn`) - rust-lang#157670 (Rename `errors.rs` file to `diagnostics.rs` (4/N)) - rust-lang#157691 (Move symbol hiding code to a new file) - rust-lang#157700 (Rename `errors.rs` file to `diagnostics.rs` (5/N)) - rust-lang#157703 (Fix doc link to Instant sub in saturating caveat) Failed merges: - rust-lang#157699 (Arg splat experiment - hir FnDecl impl)

JonathanBrouwer · 2026-06-13T10:49:54Z

@rust-timer build d97e782

rust-timer · 2026-06-13T14:40:57Z

Finished benchmarking commit (d97e782): comparison URL.

Overall result: ❌ regressions - no action needed

Benchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up.

@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.6%	[0.6%, 0.6%]	1
Regressions ❌ (secondary)	0.2%	[0.2%, 0.2%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.6%	[0.6%, 0.6%]	1

Max RSS (memory usage)

Results (primary -1.9%, secondary -2.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.8%, 0.8%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.7%	[-3.5%, -2.0%]	3
Improvements ✅ (secondary)	-2.2%	[-2.2%, -2.2%]	1
All ❌✅ (primary)	-1.9%	[-3.5%, 0.8%]	4

Cycles

Results (primary 2.1%, secondary -4.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.1%	[2.1%, 2.1%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.2%	[-4.2%, -4.2%]	1
All ❌✅ (primary)	2.1%	[2.1%, 2.1%]	1

Binary size

Results (primary 0.2%, secondary 0.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.0%, 0.6%]	16
Regressions ❌ (secondary)	0.5%	[0.4%, 0.6%]	42
Improvements ✅ (primary)	-0.1%	[-0.1%, -0.1%]	1
Improvements ✅ (secondary)	-0.0%	[-0.0%, -0.0%]	1
All ❌✅ (primary)	0.2%	[-0.1%, 0.6%]	17

Bootstrap: 517.006s -> 516.074s (-0.18%)
Artifact size: 401.43 MiB -> 400.91 MiB (-0.13%)

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Apr 19, 2026

rustbot assigned Mark-Simulacrum Apr 19, 2026

rustbot added the A-Unicode Area: Unicode label Apr 19, 2026

This comment has been minimized.

Sign in to view

Jules-Bertholet force-pushed the riir-printable branch from c2aeaba to fd13759 Compare April 19, 2026 22:55

This comment has been minimized.

Sign in to view

Jules-Bertholet force-pushed the riir-printable branch 2 times, most recently from ab09b17 to a799ecf Compare April 20, 2026 00:59

Mark-Simulacrum added the I-libs-api-nominated Nominated for discussion during a libs-api team meeting. label Apr 26, 2026

Mark-Simulacrum requested changes Apr 26, 2026

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 26, 2026

Jules-Bertholet force-pushed the riir-printable branch from a799ecf to 6079a98 Compare April 26, 2026 22:40

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 26, 2026

Amanieu removed the I-libs-api-nominated Nominated for discussion during a libs-api team meeting. label May 5, 2026

rust-rfcbot added proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. labels May 5, 2026

Mark-Simulacrum approved these changes May 9, 2026

View reviewed changes

Jules-Bertholet added 4 commits June 9, 2026 00:22

Improve core::char::mathods.rs docs

df53132

And rename a struct field.

char: move is_numeric next to is_alphanumeric

c15495f

Consider all Default_Ignorable_Code_Points unprintable

4db9ff5

These characters may be hidden/invisible otherwise.

Jules-Bertholet force-pushed the riir-printable branch from 6079a98 to 4db9ff5 Compare June 9, 2026 04:36

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 9, 2026

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 11, 2026

jhpratt mentioned this pull request Jun 11, 2026

Rollup of 28 pull requests #157735

Closed

jhpratt mentioned this pull request Jun 11, 2026

Rollup of 31 pull requests #157739

Merged

rust-bors Bot merged commit 0fb1950 into rust-lang:main Jun 11, 2026
12 checks passed

rustbot added this to the 1.98.0 milestone Jun 11, 2026

Jules-Bertholet deleted the riir-printable branch June 11, 2026 11:29

This comment has been minimized.

Sign in to view

Uh oh!

Conversation

Jules-Bertholet commented Apr 19, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Apr 19, 2026

Uh oh!

rustbot commented Apr 19, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

Mark-Simulacrum commented Apr 26, 2026

Uh oh!

Mark-Simulacrum left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustbot commented Apr 26, 2026

Uh oh!

Jules-Bertholet commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nia-e commented Apr 28, 2026

Uh oh!

Manishearth commented Apr 28, 2026

Uh oh!

Manishearth commented Apr 28, 2026

Uh oh!

Amanieu commented May 5, 2026

Uh oh!

rust-rfcbot commented May 5, 2026 • edited by dtolnay Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mark-Simulacrum left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustbot commented Jun 9, 2026

Uh oh!

Jules-Bertholet commented Jun 9, 2026

Uh oh!

Mark-Simulacrum commented Jun 11, 2026

Uh oh!

rust-bors Bot commented Jun 11, 2026

Uh oh!

Uh oh!

JonathanBrouwer commented Jun 13, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jun 13, 2026

Overall result: ❌ regressions - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Jules-Bertholet commented Apr 19, 2026 •

edited by rustbot

Loading

Mark-Simulacrum left a comment •

edited by rustbot

Loading

Jules-Bertholet commented Apr 26, 2026 •

edited

Loading

rust-rfcbot commented May 5, 2026 •

edited by dtolnay

Loading

Mark-Simulacrum left a comment •

edited by rustbot

Loading