Parser library for Kotlin consisting of a tokenizer and expression parser.
Tokenization is the process of splitting the input into a stream of tokens that is consumed by a parser.
In Parsek, most parsers only need to interact with Scanner (and optionally RegularExpressions); the underlying Lexer is exposed as a raw Iterator<Token<T>> for cases that need custom token-stream processing.
The Scanner class (source, kdoc) provides a convenient API over a token stream:
- It can be constructed directly from an input string and
Regex to tokenTypepairs (mapping skipped tokens such as whitespace tonull), or over anyIterator<Token<T>>. - Subclassing
Scanner(e.g.class Pl0Scanner(input: String) : Scanner<TokenType>(input, TokenType.EOF, ...)) is recommended so parser methods can acceptPl0Scannerwithout repeating<TokenType>, and custom helper methods can be added if needed. DirectScanner(...)instantiation is also fine for simple cases. - Store custom
Regexinstances in constants (private valorRegularExpressions) rather than constructing them inside theScannerconstructor on each parse call. - Passing a non-null
eofType(e.g.TokenType.EOF) keepsToken.typenon-nullable. To avoid adding anEOFentry to an enum, passnullwith a nullable type parameter (Scanner<TokenType?>(input, null, ...)), or useScanner<Unit?>(input, null, ...)(mapping emitted tokens toUnit) when you don't need token types at all. - It maintains a
currenttoken that can be inspected multiple times (unlikeIterator.next()), unlimited dynamiclookAhead(n), andconsume/tryConsume/consumeIgnoreCase/tryConsumeIgnoreCasehelpers.
Typical use cases that only need a scanner and no expression parser are data formats such as JSON or CSV.
For a simple example, please refer to the JSON parser example.
The RegularExpressions object (source, kdoc) contains precompiled regular expressions for common source code and data format tokens.
The token class (source, kdoc) stores the token type (typically a user-defined enum), the token text and the token position.
The lexer (source,
kdoc) implements Iterator<Token<T>>, splitting an input string into tokens using Regex to tokenType pairs (multiple regular expressions may map to the same token type, and rules mapped to null are skipped).
You only need to instantiate or subclass Lexer directly when installing additional processing between raw tokenization and Scanner — for instance via lexer.asSequence().filter { ... }.iterator() for context-sensitive newline filtering in indentation-based languages.
The Pratt expression parser (source, kdoc) operates on a scanner, is stateless and should be shared / reused.
- For ternary expressions, create a suffix expression and use the supplied scanner to consume the rest of the ternary.
- Functions / "Apply" can be implemented in a similar way. Alternatively, this can be implemented in primary expression parsing by checking for an opening brace after the primary expression.
- "Grouping" brackets should be implemented where primary expressions are processed, too.
-
A simple example evaluating mathematical expressions directly (opposed to building an explicit parse tree) can be found in the tests
-
A complete PL/0 parser is included in the examples module to illustrate how to use the expression parser and scanner for a simple but computationally complete language: Pl0Scanner.kt, Pl0Parser.kt, Pl0Test.kt
-
A parser for mathematical expressions: ExpressionLexer.kt, ExpressionScanner.kt, ExpressionParser.kt, ExpressionsTest.kt
-
A simple example for using the lexer,
Sequence.filter, scanner, and expression parser to implement an indentation-based programming language: mython, MythonTest.kt -
A BASIC interpreter using Parsek: https://github.com/stefanhaustein/basik
- Simplified
Lexer(Iterator<Token<T>>) rules toPair<Regex, T?>directly, removing the mapping lambda andtextNormalization. - Changed
ScannerEOF token text to"<EOF>", addedconsumeIgnoreCase(...)/tryConsumeIgnoreCase(...), and deprecatedFilter, 3-argumentScanner(input, eofType, eofText), andignoreCaseboolean overloads. - Removed
ConfigurableExpressionParser(org.kobjects.parsek.expressionparser). - Renamed
RegularExpressions.NUMBERtoUNSIGNED_NUMBER(withNUMBERdeprecated for one release) and fixedRegularExpressions.SYMBOLescaping for JS/Wasm. - Made
Operatorasealed class, addedPrattParser<S, Unit, R>.parseExpression(scanner), and dropped empty Javadoc JARs.
- Short-lived
Lexerfactory experiment that brokeScannersubclassing; do not use.
- Deprecated
ConfigurableExpressionParserin favor ofPrattParser. - Migrated publishing to Maven Central Portal and updated Gradle / Kotlin / Dokka toolchain.
- Added
PrattParser(org.kobjects.parsek.expression). - Added
Scannerconvenience constructor accepting regular expression pairs directly.
- Added
Filterbase class and expandedRegularExpressions. - Added the
mythonindentation-based language example.
- Enabled
wasmJstarget.
- Package name cleanup (
org.kobjects.parsek), Maven Central publishing support, and JSON parser example.