Delete one grapheme cluster in backdelete and delete

[taladar]

LLM Contributions: Claude

Backspace and forward delete should each remove one grapheme cluster — the unit a reader perceives as a character. Both removed less, so a single perceived character took several presses to erase.

backdelete deleted the whole upstream cluster only for a hard line break or an emoji cluster, and otherwise fell back to char_indices().next_back(), i.e. one codepoint. delete had no special case at all: it removed the downstream layout cluster, which comes from shaping and so tracks the font's glyphs rather than perceived characters — splitting a grapheme wherever the font does not ligate the sequence, and splitting a CRLF pair in two.

Presses to erase a single grapheme, before → after:

input	backdelete	delete
👨‍👩‍👧‍👦 ZWJ family	7 → 1	7 → 1
🇯🇵 regional-indicator flag	2 → 1	2 → 1
❤️ U+2764 U+FE0F	2 → 1	2 → 1
👋🏽 waving hand + skin tone	1 → 1	2 → 1
e + combining acute U+0301	2 → 1	2 → 1
각 Hangul jamo U+1100 U+1161 U+11A8	1 → 1	3 → 1
CRLF	1 → 1	2 → 1

Note the last three rows are not emoji at all, so no amount of refining the emoji check would have fixed them: the operations were reaching for the wrong unit. Segment on grapheme-cluster boundaries instead, using the ICU grapheme segmenter already reachable via AnalysisDataSources. The is_hard_line_break() || is_emoji() special case then becomes unnecessary and is dropped — both are simply graphemes.

Add tests covering each case above in both directions, plus a guard that two separate graphemes still take two presses.

[raphlinus]

I haven't carefully looked at the backspace logic in Parley recently, but in general it should be finer granularity than grapheme clusters. We had fairly carefully worked out logic in linebender/xilem#303, as far as I'm concerned that's still a good reference.

[taladar]

Well, in the version right before that patch it seemed to just delete by codepoint which can produce very strange results for anything that actually combines them like many emoji or non-trivial characters in non-Latin alphabets do but I am certainly open for suggestions for more refined logic from people actually using some of those languages in their everyday typing.

[taladar]

Actually no, that is not quite true, I might have missed something there, let me re-check.

[DJMcNab]

Welcome!
Please review https://linebender.org/wiki/llm-policy/. Apologies that this isn't linked more prominently - it's on the todo list.

Also, as a stylistic thing, I believe that we don't want to land commits as 'co-authored-by Claude'. You are completely responsible for the LLM's output, and so the co-authored claim isn't correct.

[taladar]

I will do that, my apologies. I just didn't want to hide the LLM involvement.

I also did some further tests and comparisons and so far it seems like the code before my commit intends to delete emojis with one delete/backdelete call but fails for some reason (I did a branch in my repo that just tests a lot of cases automatically instead of relying on the manual test of "this all looks horribly wrong with the cursor in the middle of emoji and other characters" that I used before that only covered a few cases.

It also looks like the story in terms of what users expect varies from culture to culture, in particular where IME and pre-edit composition of characters is involved deleting individual codepoints isn't really that common (India seems to be an exception).

Parley also seems to be just about the only software that deletes CRLF as two individual characters with backdelete/delete.

But overall I will still need to investigate a bit further before I can update this.

[DJMcNab]

No worries! I'm glad to see that you're being thorough here.

I expect that we'll discuss this in next week's text office hours - that will probably be next Tuesday at 07:00 UTC (I believe, you should double check that on Zulip if you wish to join. We are planning a ~monthly office hours to be a more US friendly time slot). It would be great to have you there if the timezone works.
In particular, we have ongoing work to clear up a lot of our semantics, and we might have some pre-existing idea of what shape would give the most correct solution here.

This Co-Authored-By: Claude thing isn't written down anywhere, so there's absolutely no need to apologise. And the impulse to be forthcoming with LLM usage is the right one!

[taladar]

I have been digging into this a bit deeper but haven't quite gotten to the point where I want to submit any new PRs.

So far I stumbled across (but haven't investigated deeper yet)

• some of your unit tests are broken because they don't add any fonts and so they literally have no cursor positions
• is_emoji exceptions for deletion don't work because individual codepoints that make up emoji grapheme clusters are also emoji
• delete/backdelete combinations can leave behind non-printable codepoints like a ZWJ on their own
• deletion/cursor key movements/clicks all differ significantly from the vast majority of other implementations and in some cases do things (like just taking the 1/nth point for clicks into an n codepoint grapheme cluster) that are explicitly discouraged in some of the relevant standards
• on top of that the story of deletion per codepoint vs. whole grapheme expectations differs signficantly between scripts/cultures, some expect indidividual codepoint deletion, others (often the ones with IME pre-edit) expect whole grapheme cluster deletion
• parley apparently also deletes CRLF individually instead of together
• apparently expectations are also different for backdelete vs delete behavior

I did some (largely LLM using and not suitable for inclusion in the upstream repo) initial research and tests of parley behavior on main...taladar:parley:research/traversal-fact-finding but haven't had the time to verify everything in there manually yet for lack of time to work on parley exclusively instead of the project I intended to use parley in.