Including underscore in <alpha> and <alnum> is confusing

[alabamenhu]

Raku currently equates <alpha> to <:L+[_]>, and <alnum> to <:L+:N+[_]>.

The underscore seems to be included so that <alnum> is the equivalent of \w and that <alpha> is the <digit>-less version thereof.

Nonetheless, this can be confusing for users who expect those named classes to align with their general POSIX definition (even if that is a bit squishy, given it's locale based).

For reference, we do align with POSIX everywhere else, even in that <punct> is defined as <:P+:S>.

[librasteve]

According to the design docs https://github.com/Raku/old-design-docs/blob/master/S05-regex.pod#predefined-subrules

alpha

Match a single alphabetic character, or an underscore.

To match Unicode alphabetic characters without the underscore, use <:alpha>.

My interpretation of this is that in the spirit of timtowdi, raku regex alpha already has both with- and without-underscore variants. This neatly addresses the two requirements of (i) consistency in that /w and alpha are the same thing (and match both perl /w and OS expectations) and (ii) closer to the POSIX standard in that :alpha is the new improved method that engages unicode.

EDIT:

per lizmat test

$ raku -e 'say "_a" ~~ / <.:alpha> /'
｢a｣
$ raku -e 'say "_a" ~~ / <.alpha> /' 
｢_｣

now I spelunked the rakudo compiler and found this here https://github.com/rakudo/rakudo/blob/ce03d170104c92a2837fef5bcc0fbf81fd602b03/src/core.c/Match.rakumod#L222

Note: no sign of :alpha

https://www.regular-expressions.info/refunicodeproperty.html shows that Alpha is a standard Unicode shortname for the Alphabetic property. So your test is just a regular selector for that (the raku syntax is case insensitive).

I think that this whole list could use some better coverage in the docs.

---

##### / <:General_Category{$property}> /
    my $general-category-property-lookup := nqp::hash(
        "Uppercase_Letter", "Lu",
        "Lowercase_Letter", "Ll",
        "Cased_Letter", "LC",
        "Titlecase_Letter", "Lt",
        "Modifier_Letter", "Lm",
        "Other_Letter", "Lo",
        "Nonspacing_Mark", "Mn",
        "Spacing_Mark", "Mc",
        "Enclosing_Mark", "Me",
        "Decimal_Number", "Nd",
        "digit", "Nd",
        "Connector_Punctuation", "Pc",
        "Dash_Punctuation", "Pd",
        "Open_Punctuation", "Po",
        "Close_Punctuation", "Pe",
        "Initial_Punctuation", "Pi",
        "Final_Punctuation", "Pf",
        "Other_Punctuation", "Po",
        "Math_Symbol", "Sm",
        "Currency_Symbol", "Sc",
        "Modifier_Symbol", "Sk",
        "Other_Symbol", "So",
        "Space_Separator", "Zs",
        "Line_Separator", "Zl",
        "Paragraph_Separator", "Zp",
        "cntrl", "Cc",
        "Control", "Cc",
        "Format", "Cf",
        "Surrogate", "Cs",
        "Private_Use", "Co",
        "Unassigned", "Cn"
    );
    my $general-category-family-lookup := nqp::hash(
        "Letter", "L",
        "L", "L",
        "Mark", "M",
        "M", "M",
        "Number", "N",
        "N", "N",
        "Punctuation", "P",
        "punct", "P",
        "Symbol", "S",
        "S", "S",
        "Separator", "Z",
        "Z", "Z",
        "Other", "C",
        "C", "C"
    );

[alabamenhu]

Indeed, <:alpha> exists, but as a consequence of the Unicode selectors.

However, in the best of technicalities, POSIX is slightly different than a pure Unicode selector, in that it can be locale dependent (hence Intl::Regex::CharClass).

Perl started supporting POSIX classes the same year that Raku (then Perl 6) was announced. In it, \w and alnum are in fact distinct, the latter not including the underscore. Given I don't tend tend to see POSIX selectors as often in Perl, I genuinely wonder if this was simply an oversight that has managed to persist decades. As you've seen in the design docs, \w was explicitly equated to <alnum>, so logically, I think, they decided that <alpha> should be <alnum> less <digit>.

It's actually enough of a trap that when someone posted about it on Reddit, I had to triple check it myself (and I see in Intl::Regex::CharClass I didn't think to put one in). That we would provide all of the POSIX classes, but then treat one of them substantially differently is a huge trap.

Obviously, it would technically be a breaking change, but looking through Rakudo source code, there are only a few instances where <alpha> is used and we would anticipate underscore to be used (like three). A bigger issue would be the ecosystem, but I imagine in 99% of cases, when folks are using <alpha>, I can guarantee they are anticipated <:L> not <:L+[_]>.

[librasteve]

@alabamenhu - you make some good points, and you are more expert than I. Thanks for confirming my analysis of the situation.

However, you do not explain why the design docs are so clear on this point:

To match Unicode alphabetic characters without the underscore, use <:alpha>.

Hopefully, you would agree that it is common for OS and application code in general to want to use alphanumunder? I can remember that this was a pain in perl and I (for one) was pleased to read the synopsis that Raku includes it as the default.

I do not think that we can just make a breaking change by appeal to the argument that if the compiler does not rely on a feature, it is therefore unlikely to break things out there. Honestly I think that matching with/without underscore is both quite common (eg. identifiers, urls, user input) and quite subtle so that things could break without warning. I'm not so sure that this is something you can guarantee.

Do you propose to break /w and /W also? Or will <alpha> no longer be consistent with /w?

If a concensus emerges to do this change then I propose that we add <alphun> and <alunum> for the underscore cases and warn on use of <alpha> and <alnum> for a period.

[alabamenhu]

My proposal would be that we align with most other languages:

Code	Meaning
\w	<:L+:N+[_]>
\W	<-:L-:N-[_]>
<alpha>	<:L>
<alnum>	<:L+:N>

In this way, \w and \W are unmodified, but <alpha> and <alnum> would be changed to align with standard usage.

Because they are just defined as tokens, I reckon this can be easily version guarded: in 6.e (or 6.f), the definition of the token simply changes, and those wanting old behavior can use the previous version. It would only require a single change to roast (afaict, it's only tested in S05-mass/charsets.t), with the rest requiring just updating Rakudo to use <:L+:N+[_]> when that's what we want, or to add in an <alphau> and <alnumu>.

Obviously, realistically it's quite easy for people to write their own token as I mentioned in the Reddit post to get their desired behavior (thus keeping alive TIMTOWTDI). The more I think about it, the more important I think we need to defer to POLA and have the default to be the one that aligns with the standard usage.

[librasteve]

@alabamenhu … this looks like a good proposal to me, and you make a good point are imo.

Based on the synopsis quote above and a general desire to avoid breaking changes, my vote is to maintain the status quo, but I see that there are two votes for change ( you and the OP on Reddit).

Suggest we see how the issue evolves and how others’ votes fall.

[librasteve]

just digging around to find evidence for/against the current design being deliberate or (as set out by @alabamenhu, an oversight).

From https://github.com/Raku/old-design-docs/blob/master/S05-regex.pod#simplified-lexical-parsing-of-patterns

Unlike traditional regular expressions, Perl 6 does not require you to memorize an arbitrary list of metacharacters. Instead it classifies characters by a simple rule. All glyphs (graphemes) whose base characters are either the underscore (_) or have a Unicode classification beginning with 'L' (i.e. letters) or 'N' (i.e. numbers) are always literal (i.e. self-matching) in regexes.

This is more evidence that, deeply, raku consistently considers _ to be alphanum alongside L or N.

[lizmat]

FWIW, I'm getting the feeling this is more a documentation issue than anything else.

Perhaps @TimToady would like to chime in on this.

[lizmat]

Also: perhaps some clarity could be obtained from https://irclogs.raku.org/search.html?query=underscore&type=words&oldest-first=True&nicks=&channel=perl6 ?

[jubilatious1]

Raku currently equates <alpha> to <:L+[_]>, and <alnum> to <:L+:N+[_]>.

I'm actually in favor of leaving things just the way they are.

Command-line coders want brief code, and <alpha>/<alnum> give them a way to briefly/concisely demarcate file_names and interesting_code_fragments. Especially when grappling with code from lesser languages!

All kidding aside, @alabamenhu 's excellent code equivalences should go straight into the docs (thanks @tbrowder and @arkiuat). Best Regards.

[ShimmerFairy]

It just so happens that a couple weeks ago I was going through all of Raku's predefined token classes, to see how it lines up with UTS#18, which offers a description of how Unicode regexes should work. I stopped working on the document I was writing because there's so much to Raku's Unicode support that could do with a once-over, and I could feel myself about to sink too much time into it considering other stuff I wanted to do.

First of all, to my eyes Raku does not align itself to POSIX at all, rather to UTS#18's "compatibility rules" section: https://www.unicode.org/reports/tr18/#Compatibility_Properties . As that section describes, aligning with POSIX is generally a bad idea, for example <digit> would not be allowed to match more than 20 characters.

Your example of <punct> shows that the documentation is wrong actually. UTS#18 recommends a definition of <:P> (punctuation category characters), and all Rakudo backends agree with this. UTS#18's POSIX-compatible definition is <:P + :S - alpha> (note: the UTS#18 definition of <alpha>, not ours), but specifically recommends against this because counting symbols as punctuation can be confusing. Additionally, S05 corroborates the idea that it should be punctuation only:

• punct

Match a single punctuation character (that is, any character from the Unicode General Category "Punctuation").

The online documentation is simply wrong, it should say <:P> is the definition.

So to that point, we don't abide by POSIX definitions. As mentioned, they're locale dependent, which we don't have support for in general, and true POSIX adherence would render us with a very bad set of grammar rules. For example, the current version of POSIX states that <digit> will only ever match <[0..9]>, which is obviously not what anybody wants. We already aren't, and won't, align the rules with the POSIX meanings of those words, so an argument based solely on that isn't very convincing.

As for the actual point of this issue, I am at this moment strongly opposed to removing underscore from the definition of <alpha> and all things that transitively include it. I've definitely relied on this in plenty of code I've written over the decades, even if most of it is personal stuff that would never show up in an ecosystem check. It must be stated that the inclusion of the underscore wasn't a mistake, it was an intentional decision to make grammars a bit easier to write. Parsers are usually written for programming languages, and programming languages usually consider the underscore to be an alphabetic character. So having it be counted as one by default makes sense.

In retrospect, should it have ever been included? Perhaps not, but it was an intentional decision whose change would definitely break code, and I'm not sure it's worth it when anyone could easily add a token alpha { <:Alpha> } to their own grammars to fix it. (There may be some Rakudo bugs where it mistakenly uses the default definition where it shouldn't, but those are bugs, and ought to be fixed.) I'm not totally opposed to breaking changes (as I discuss below), but by itself it would just be a change that breaks existing code for no real benefit.

The funny thing though is that there is a problem with <alpha>, but not the one brought up here. Currently, Rakudo (except on JVM, where things get a bit weird) uses <:L> to match alphabetic characters, whereas UTS#18 recommends <:Alphabetic> instead. (Indeed, the MoarVM code includes an XXX note asking if this should be changed.) However, changing this could have knock-on effects on what Rakudo considers valid identifier characters, which would likely require us to divorce identifier parsing from the general <alpha> and <alnum> rules, and consider UAX#31: Unicode Identifiers and Syntax for how to better define identifiers specifically.

Finally, the \w matcher is mentioned as related to these concerns. I'll point out that UTS#18 has separate definitions for <alnum> and <word> (a rule we don't have), and \w matches <word> in their definitions. Raku has, up to this point, decided to unify "alphanumeric character" and "word character", with potentially confusing results (judging by the NQP and MoarVM source code). The issue of what to do with \w is its own can of worms. Funny enough, <word> includes the underscore, and some other, "Unicode-ier" underscores for good measure.

In short: I think removing underscore from <alpha> isn't a worthwhile breaking change for what ultimately amounts to temporary confusion in new developers, at least not without some kind of new default rule for "I want to treat underscores like letters" to take its place. If we're gonna modify one rule, we should go over them all, and they ought to line up with UTS#18. (And forget about strict adherence to POSIX ctypes; we're never gonna do that in the default grammar.)

A final note: if people are interested, I can get back to my document going over all the predefined rules, and how their current definitions line up with various bits of documentation. Considering the whole problem at once will likely lead to better results than fixing one issue in one corner of the default grammar. (The document would go over all the UTS#18 compatibility rules, as well as how UAX#31 and <ident> should interact.)

[alabamenhu]

The locale-awareness isn't a huge issue for us: in general, we do things assuming the c/root/default locale and generally leave it to module space for anything beyond that. [1] I think there's still some stubbed code for locale-based collation somewhere in MoarVM or NQP.

I did notice as I was investigating further that, contrary to docs, we do define punctuation as <:P> (probably should have edited). This causes some surprises for people in other languages like Ruby that distinguish ASCII and Unicode parsing. In Ruby with ASCII, < parses as :punct: (because in ASCII, :punct: is :graph: - :alnum:) but the latter '<' does not, because it uses Unicode definitions and < is defined as a symbol. That's definitely a trap when moving outside of ASCII, as you note the difference between <:P>, <:P+:S-:Alphabetic>. Speaking of, good catch on <:L> vs <:Alphabetic>. I'd agree that that change should be made (but only at the same time as any other wider changes.)

I'm actually not opposed to us reviewing all of the rules. Like you, I agree we have a lot of work Unicode-wise that needs to be done. It's probably easier to do it all at once rather than piecemeal.

---

[1] I say this as the person who once long ago pushed for some locale-aware bits in core. I've changed my mind on that and now fully support keeping that in module-space, and I'm (likely) the person who has spent the most time advancing locale-based stuff in Raku. That said, Unicode might be the one place we do want to provide some capability, given just how ugly putting that into module space would be, if even possible. That's a different discussion for a different day, though.