A lexer, a PEG parser, a grammar compiler and an error printer for PHP. Describe your format once — in a file that reads like the language it describes — and get tokens, a tree, and errors that point at the character.
composer require phplrt/phplrt
// the words %skip T_WHITESPACE \s++ %token T_DIGIT \d++ %token T_PLUS \+ // the sentences, and what to build out of them Sum -> { return \array_sum($children); } : Number() (::T_PLUS:: Number())* ;
The lexer turns characters into tokens. The parser checks that those tokens appear in an order the grammar allows, and builds whatever you asked for. Nothing else happens in between.
A string, a file or a stream — wrapped in an object that knows its content and what to call itself when an error has to say where.
new File(__DIR__ . '/example.txt');
new VirtualFile('config.txt', $text);
new Source('2 + 2');
new Stream($resource);
Every token carries id, name,
value, offset and size — offsets
in bytes, which is what makes snippets possible later.
Rules live in a table, not in a call graph, and each one knows the tokens it may start with. Recognition writes a flat trace; the reducers run once, afterwards, bottom-up. A branch that fails is simply truncated.
Config ├── Pair │ ├── T_NAME "name" │ └── T_STRING "phplrt" └── Pair ├── T_NAME "version" └── T_NUMBER "4"
Whitespace and comments are hidden, not skipped: the parser does
not see them, a documentation generator can. Unrecognized text becomes an
Unknown token instead of an exception, so a linter reports a
whole file rather than its first problem.
Alternatives are tried top to bottom and the first match wins, so a
grammar is never ambiguous. The flip side: "a" | "ab" never
reads ab — put the longer one first.
backtracking PEG recognizer · table-driven recursive descent · FIRST-set prediction · deferred tree construction how the parser works
A grammar file says what the words are and what order they may come in. Everything else — states, channels, reducers — is an optional suffix on one of these two lines.
*: means every state
T_QUOTE
the name — for you and for error messages. The parser compares ids
"
the expression. No literal spaces: write \x20
-> state(strings), channel(quotes)
what the token does besides being read: hand the reading over to another lexer, and put itself on a channel of its own
Field()
-> { return new Node($offset, $children); }
the reducer. Gets whatever the body matched and returns whatever you want — a number, a node, an array. Omit it and the tokens come through as they are
: <T_NAME>
a token, kept
::T_COLON::
a token, discarded
Value()? ;
another rule, optional. Also *, +, {2,5}, and &e / !e to look ahead without reading
The lexer takes the first pattern that matches. Declare **
before *, and keywords before the identifier pattern — or
if becomes a name.
Expr : Expr() "+" Number() is refused while building, not at
runtime. Write it as a repetition — which is what the diagram in the hero
shows anyway.
"=" inside a rule declares its own token, and such a token is
always discarded. /and|or/ does the same with a regular
expression.
A grammar is normally shorter than the code you would write to read the same thing by hand — and unlike that code, it can be drawn.
%skip T_WHITESPACE \s++ %skip T_COMMENT //[^\n]*+ %token T_NUMBER \d++(?:\.\d++)? %token T_STRING "[^"]*+" %token T_TRUE true %token T_FALSE false %token T_NAME [a-zA-Z_][a-zA-Z0-9_]*+ %pragma root Config Config : Pair()* ; Pair : <T_NAME> "=" Value() ; Value : <T_NUMBER> | <T_STRING> | List() ; List : "[" (Value() ("," Value())*)? "]" ;
name = "phplrt" version = 4 tags = ["php", "peg"] // comment is hidden
%skip T_WHITESPACE \s++ %token T_STRING "[^"]*+" %token T_NUMBER \d++(?:\.\d++)? %token T_AND and %token T_OR or %token T_FIELD [a-z_][a-z0-9_.]*+ %pragma root Query // "or" binds looser, so it is read first and // written in terms of "and" — precedence is nesting Query -> { return Any::of($children); } : All() (::T_OR:: All())* ; All -> { return Every::of($children); } : Test() (::T_AND:: Test())* ; Test : Compare() | "(" Query() ")" ; // "<=" before "<": first match wins, not longest Compare : <T_FIELD> ("<=" | ">=" | "<" | ">" | "=") Val() ; Val : <T_STRING> | <T_NUMBER> ;
price < 100 and brand = "acme"
stock > 0 or (rating >= 4 and reviews > 10)
"<=", "(" and friends declare their tokens
where they are read, and an inline token is always discarded — five
%token lines saved, none of which needed a name.
The moment a language nests, a regular expression stops being able to describe it — and the hand-written reader that replaces it is exactly the code this grammar is instead of.
// inside {{ … }} the rules are different, so the // opening token hands the reading to another lexer %skip *:T_WHITESPACE \s++ %token T_TEXT [^{]++ %token T_OPEN \{\{ -> state(code) %token code:T_NAME [a-z_]\w*+ %token code:T_DOT \. %token code:T_CLOSE \}\} -> exit() // kept out of the grammar, but not thrown away %token T_DOC /\*\*.*?\*/ -> channel(docblocks) // and when no regex will do — write the lexer %token T_PHP <\?php -> state(php) %lexer php -> { new \App\Lexer\PhpTokenLexer() } %pragma root Template Template : (<T_TEXT> | Hole())* ; Hole : <T_OPEN> ;
Everything the inner lexer read is carried by the token that entered
it: $token->children is the inner stream and
$token->size covers all of it. The grammar only ever
names <T_OPEN>.
channel(docblocks) hides a token from the parser and
leaves it in the stream — your doc generator reads the same pass.
the docs ship complete grammars: math, calculator, json, json5, phpdoc types, go! aop pointcuts browse
“Syntax error at offset 137” is technically correct and practically useless. Four things can go wrong while reading a source — all four print the same way.
A bare Source can only show the snippet.
new VirtualFile('input.txt', $text) costs nothing and lets
every error above say where.
error[UnexpectedTokenException]: Syntax error, unexpected "and" (T_AND) filter.txt:1:24 price < 100 and brand in and ("acme") ^^^ = the position reported is the furthest one any rule reached, not where the last attempt gave up
use Phplrt\Parser\Exception\UnexpectedTokenException; use Phplrt\Source\VirtualFile; try { $parser->parse(new VirtualFile('filter.txt', $input)); } catch (UnexpectedTokenException $e) { echo $e->getMessage(); // the sentence echo $e; // …and the block above $e->token->name; // T_AND $e->token->offset; // 23 }
warning: Unrecognized input, skipped filter.txt:1:13 price < 100 § and stock > 0 ^ = the lexer never stops on strange input: "§" arrives on the Unknown channel, so one pass reports a whole file instead of its first problem
use Phplrt\Contracts\Lexer\Channel; use Phplrt\Exception\ErrorPrinter; use Phplrt\Exception\Printer\Level; foreach ($lexer->lex($source) as $token) { if ($token->channel !== Channel::Unknown) { continue; } echo new ErrorPrinter() ->print($source, $token->offset, $token->size) ->withMessage('Unrecognized input, skipped') ->withLevel(Level::Warning) ->withLinesAround(0); }
error[UnknownFieldException]: Field "brnd" does not exist filter.txt:1:17 price < 100 and brnd = "acme" ^^^^ = did you mean "brand"?
final class FieldException extends \RuntimeException { public function __construct( string $message, public readonly ReadableInterface $source, public readonly int $offset, public readonly int $length = 0, ) { parent::__construct($message); } public function __toString(): string { return (string) new ErrorPrinter() ->print($this->source, $this->offset, $this->length) ->withMessage($this->getMessage()) ->withClass(static::class); } }
// "price < 100 and" — someone is still typing PartialResult read a fragment and stopped value Every(Cond(price, <, 100)) token "and" at offset 12 diagnostics[0] — the exception parse() would have thrown filter.txt:1:13 price < 100 and ^^^
use Phplrt\Parser\Analysis\Mode; use Phplrt\Parser\Analysis\Result\SuccessfulResult; $result = $parser->analyze($source); // SuccessfulResult · PartialResult · FailureResult $result->value; // what the fragment reduced to $result->token->offset; // where it stopped $result->diagnostics; // and what stood in the way // greedy rules tell "not finished" from "written wrong" — // which is exactly what a search box needs if ($parser->analyze($typed, Mode::SyntaxCheck) instanceof SuccessfulResult) { $button->enable(); }
One renderer, three levels, colours that follow the output stream.
withLinesAround(0) when a linter has forty of them.
PrinterInterface takes the captured lines and returns a
string: SARIF, LSP diagnostics, GitHub annotations, one line per error
for a CI log.
SourceExceptionInterface, LexerExceptionInterface,
ParserExceptionInterface — catch as coarsely or as finely as
the situation deserves.
the three interfaces ship with phplrt/runtime · the printer is phplrt/exception, installed separately
printing errors
Under a kilobyte every one of these reads a type expression in about a
millisecond and the difference between them is noise. It is the hundredth
kilobyte that separates them: phplrt costs ×8.8 for every ×10 of input,
which is as close to linear as reading gets.
phpstan/phpdoc-parser costs ×78 — and at a quarter of a
megabyte that is 24 seconds against 194 ms.
| tool | 25 B | 250 B | 1 KB | 10 KB | 100 KB | 250 KB | per ×10 of input |
|---|---|---|---|---|---|---|---|
| phplrt 4.x | 0.81 | 1.15 | 1.83 | 8.92 | 78.1 | 194 | ×8.8 |
| phplrt 3.x | 0.85 | 1.74 | 4.55 | 35.1 | 335 | 901 | ×9.6 |
| hoa/compiler | 0.85 | 2.79 | 9.99 | 87.1 | 877 | 2 353 | ×10.1 |
| vimeo/psalm | 0.68 | 1.29 | 2.28 | 16.9 | 722 | 5 154 | ×43 |
| phpstan/phpdoc-parser | 0.70 | 1.02 | 2.15 | 47.9 | 3 725 | 24 177 | ×78 |
| tool | 25 B | 250 B | 1 KB | 10 KB | 100 KB | 250 KB | × ours at 250 KB |
|---|---|---|---|---|---|---|---|
| phplrt 4.x | 0.81 | 0.83 | 0.92 | 1.81 | 11.9 | 28.6 | 1.0× |
| vimeo/psalm | 1.57 | 1.61 | 1.79 | 3.63 | 21.0 | 49.2 | 1.7× |
| phpstan/phpdoc-parser | 0.78 | 0.81 | 1.01 | 2.86 | 20.4 | 55.2 | 1.9× |
| phplrt 3.x | 0.83 | 0.87 | 1.10 | 3.28 | 24.0 | 60.8 | 2.1× |
| hoa/compiler | 0.95 | 1.31 | 2.64 | 15.7 | 139 | 336 | 11.8× |
(*MARK:n), one pass over the input
Almost every public API changed — property hooks, asymmetric visibility
and new in initializers are used throughout, which is why so
much of it looks different. Grammar files mostly compile unchanged.
The API is built on features that only exist there, which is also what made the lookahead tables and the code generation practical.
A generated parser is a real class again — the lexer, the token constants and the reducers as methods. Commit it and the compiler stays a dev dependency.
No execution context between calls, so one parser is safe to share and to call from several places at once.
grammar files carry over · the constructors do not upgrade guide
The compiler reads your grammar and writes a parser — a development job. The runtime reads your users' text. Keep them apart and production carries very little.
Read the grammar, generate the parser, commit the result. Re-run it when the grammar changes — which, at the start, is every five minutes.
phplrt/compilerreads .pp3 / .pp2, resolves %include, writes PHPphplrt/lexer-buildera lexer described in PHPphplrt/parser-buildera grammar described in PHP$ composer require phplrt/compiler --dev $ vendor/bin/phplrt compile grammar.pp3 src/RouteParser.php \ --namespace='App\Routing' \ --class=RouteParser
The generated parser is plain PHP referring to the runtime — no grammar file, no compilation, nothing to warm up.
phplrt/sourcefiles, strings, streams — with namesphplrt/lexertext into tokensphplrt/parsertokens into your result· 3 contractsthe source, lexer and parser interfaces the three are written againstphplrt/exceptionoffsets into snippets — a separate package, add it if you print errors$ composer require phplrt/runtime
php 8.4+ · pcre · mbstring recommended · mit installation guide
A search filter, a routing table, a docblock type, an in-house format nobody wrote a package for. Half an hour to a grammar, and every mistake in it gets an error that points at the character.
composer require phplrt/phplrt