`String.Parser.keyword` treats every non-ASCII letter as a word boundary; compiler-common's parser thus disagrees with the Haskell-based parser

[gilramir]

Found against: gren 0.6.6, gren-lang/core 7.4.2, node 22

Code to reproduce is in this git repo: https://github.com/gilramir/gren-bug-reports
in the 2026-09-07-non-ascii-tokens directory.

String.Parser.keyword

A keyword parser has to refuse to match when the keyword is only the front of a
longer name. In letters the let is not a keyword — it is the first three
letters of a variable. token "let" matches both; String.Parser.keyword
exists to tell them apart, and its docs say so:

Note: this would fail to chomp letter because of the subsequent
characters. Use token if you do not want that
last letter check.

It makes that decision by looking at the one character after the keyword and
asking whether that character could be part of a name. If it could, this is not
a keyword and the parser fails. That test is the word boundary. It is not
"whitespace" and not "punctuation" — it is "the next character is not a letter,
digit or underscore".

I could not find anything in any language spec declaring that non-ASCII characters
can be in a keyword, but the Haskell-based parser accepts them, but String.Parser.keyword,
which compiler-common uses, does not.

Non-ASCII characters in identifiers

This snippet comes from src/Identifiers.gren in the reproduction repo:

café : Int
café =
    1


ｚebra : Int
ｚebra =
    2


𝐚stral : Int
𝐚stral =
    4


type Týpe
    = Éins
    | Zwölf


value : Týpe -> Int
value t =
    when t is
        Éins ->
            café

        Zwölf ->
            ｚebra + 𝐚stral

With the Haskel-based compiler, it runs fine:

% devbox run gren run Identifiers
Info: Running script "gren" on /home/gram/prj/gren-bug-reports/2026-09-07-non-ascii-tokens
7

However, gren-format uses the compiler-common's Gren-based parser, which doesn't like it:

% gren-format src/Identifiers.gren
-- FAILED TO PARSE ---------------------------------------- src/Identifiers.gren

20|
21|
22| 𝐚stral : Int
    ^
23| 𝐚stral =
24|     4

Expected end of file
Expected keyword 'port'
Expected keyword 'type'
Invalid character in variable name

The bug

gren-lang/compiler-common check for keywords with
String.Parser.Advanced.keyword, which is the problem.

keyword asks Char.isAlphaNum, which is ASCII by contract. Its own docs
say "Detect upper case and lower case ASCII characters" and give
isAlphaNum 'π' == False as an example. So every letter outside ASCII looks
like the end of the keyword, and keyword "let" matches the front of a name
that continues into one.

The caller cannot correct it from outside: the boundary predicate is not a
parameter. And Char.isAlphaNum is the right function for what it says it does
— the mismatch is that keyword uses it to mean "part of a name", which is a
decision belonging to the language being parsed rather than to ASCII.

Reproduction

$ ./run.sh

src/Boundary.gren — keyword on its own

keywordLet : String -> String
keywordLet source =
    when P.run (P.succeed {} |> P.skip (P.keyword "let")) source is
        Ok _ ->
            "Ok "

        Err _ ->
            "Err"


main : Node.SimpleProgram a
main =
    Node.defineSimpleProgram (\env -> Node.endSimpleProgram (emit env))


emit : Node.Environment -> Task Never {}
emit env =
    [ "letters", "let\u{00E9}s", "let\u{FF5A}s", "let\u{1D41A}s" ]
        |> Array.map (\source -> keywordLet source ++ "   " ++ source)
        |> String.join "\n"
        |> (\text -> Stream.writeLineAsBytes text env.stdout)
        |> Task.map (\_ -> {})
        |> Task.onError (\_ -> Task.succeed {})

It runs keyword "let" against four names and prints Ok or Err for each.
All four are single names — a variable called letters, a variable called
letés, and so on. No keyword occurs in any of them, so Err is correct
for all four.

Look at the "=== src/Boundary.gren" section in the output.

Err   letters
Ok    letés
Ok    letｚs
Ok    let𝐚s

source	character after let		correct	actual
letters	s	U+0073, ASCII	Err	Err
letés	é	U+00E9, 2 UTF-8 bytes	Err	Ok
letｚs	ｚ	U+FF5A, 3 UTF-8 bytes	Err	Ok
let𝐚s	𝐚	U+1D41A, 4 UTF-8 bytes	Err	Ok

Ok means the parser took let to be a whole keyword and consumed it, leaving
és behind for whatever the grammar expects next.

The three widths are deliberate — two UTF-8 bytes, three, four — and they all
behave the same way, which is the point: this is about Char.isAlphaNum's
answer, not about the encoding.

[robinheghan]

Fixed in release branch