phplrt is a parsing toolkit for PHP: a lexer, a PEG parser, a grammar compiler and an error printer. You describe your format once, in a file that reads much like the language it describes, and you get back tokens, a syntax tree and error messages that point at the exact character.
composer require phplrt/phplrt
// the tokens %skip T_WHITESPACE \s++ %token T_DIGIT \d++ %token T_PLUS \+ // the rules, and what to build out of them Sum -> { return \array_sum($children); } : Number() (::T_PLUS:: Number())* ;
The lexer turns characters into tokens. The parser then checks that those tokens appear in an order the grammar allows and builds the result you asked for. There are no other stages in between.
A string, a file or a stream, wrapped in an object that knows both its content and its own name. The name is what an error message uses to say where the problem is.
FileSource::createFromPathname(
pathname: __DIR__ . '/example.txt',
);
StringSource::createFromString(
content: '2 + 2',
);
ResourceSource::createFromResource(
resource: $resource,
);
Every token carries id, name,
value, offset and size. Offsets are
counted in bytes, which is what lets an error printer cut out the right
snippet later.
Rules are kept in a table rather than compiled into nested function calls, and each rule knows which tokens it may start with. Recognition writes a flat trace of what matched; the reducers run afterwards, once each, from the innermost rule outwards. A branch that fails is cut off.
Config ├── Pair │ ├── T_NAME "name" │ └── T_STRING "phplrt" └── Pair ├── T_NAME "version" └── T_NUMBER "4"
Instead of throwing a token away, the lexer puts it on a channel, and it
emits every channel except the ones you tell it to skip. By default that
is Hidden alone. Give docblocks a channel of their own and
they stay in the stream that the parser reads past. Text the lexer does
not recognize arrives as an Unknown token instead of an
exception, so a linter can report a whole file rather than stopping at
the first problem.
Alternatives are tried from top to bottom and the first one that matches
wins, so a grammar can never be ambiguous. The catch is that
"a" | "ab" will never match ab, so the longer
alternative has to come first.
backtracking PEG recognizer · table-driven recursive descent · FIRST-set prediction · deferred tree construction how the parser works
One kind declares a token, the other declares a rule. Everything else, including states, channels and reducers, is an optional suffix on one of those two lines.
*: means every state
T_QUOTE
the name, used by you and by error messages. The parser itself compares ids
"
the expression. No literal spaces: write \x20
-> state(strings), channel(quotes)
what happens once the token is matched: hand the reading over to another lexer, and move the token onto a channel of its own
Field()
-> { return new Node($offset, $children); }
the reducer. It receives whatever the body matched and returns whatever you need: a number, a node, an array. Leave it out and the tokens come through unchanged
: <T_NAME>
a token, kept
::T_COLON::
a token, discarded
Value()? ;
another rule, optional. There are also *, + and {2,5}, plus &e / !e for lookahead that reads nothing
The lexer uses the first pattern that matches, not the longest one.
Declare ** before * and keywords before the
identifier pattern, or if will be read as a name.
Expr : Expr() "+" Number() is rejected when the grammar is
built, not at runtime. Write it as a repetition instead, the way the
diagram in the hero does.
Writing "=" inside a rule declares a token for it on the spot,
and a token declared this way is always discarded. /and|or/
does the same with a regular expression.
A grammar is usually shorter than the code you would write to read the same format by hand, and unlike that code it can be drawn as a diagram.
%skip T_WHITESPACE \s++ %skip T_COMMENT //[^\n]*+ %token T_NUMBER \d++(?:\.\d++)? %token T_STRING "[^"]*+" %token T_TRUE true %token T_FALSE false %token T_NAME [a-zA-Z_][a-zA-Z0-9_]*+ %pragma root Config Config : Pair()* ; Pair : <T_NAME> "=" Value() ; Value : <T_NUMBER> | <T_STRING> | List() ; List : "[" (Value() ("," Value())*)? "]" ;
name = "phplrt" version = 4 tags = ["php", "peg"] // comment is hidden
%skip T_WHITESPACE \s++ %token T_STRING "[^"]*+" %token T_NUMBER \d++(?:\.\d++)? %token T_AND and %token T_OR or %token T_FIELD [a-z_][a-z0-9_.]*+ %pragma root Query // "or" binds looser, so it is read first and // written in terms of "and": precedence is nesting Query -> { return Any::of($children); } : All() (::T_OR:: All())* ; All -> { return Every::of($children); } : Test() (::T_AND:: Test())* ; Test : Compare() | "(" Query() ")" ; // "<=" before "<": first match wins, not longest Compare : <T_FIELD> ("<=" | ">=" | "<" | ">" | "=") Val() ; Val : <T_STRING> | <T_NUMBER> ;
price < 100 and brand = "acme"
stock > 0 or (rating >= 4 and reviews > 10)
"<=", "(" and the rest declare their tokens
right where they are used, and an inline token is always discarded.
That saves five %token lines, none of which needed a name.
As soon as a format allows brackets inside brackets, a regular expression can no longer describe it. You end up writing the reader by hand instead, and that is exactly the code this grammar replaces.
// inside {{ … }} the rules are different, so the // opening token hands the reading to another lexer %skip *:T_WHITESPACE \s++ %token T_TEXT [^{]++ %token T_OPEN \{\{ -> state(code) %token code:T_NAME [a-z_]\w*+ %token code:T_DOT \. %token code:T_CLOSE \}\} -> exit() // hidden from the parser, but still in the stream %token T_DOC /\*\*.*?\*/ -> channel(docblocks) // and when no regex will do, plug in your own %token T_PHP <\?php -> state(php) %lexer php -> { new \App\Lexer\PhpTokenLexer() } %pragma root Template Template : (<T_TEXT> | Hole())* ; Hole : <T_OPEN> ;
Everything the inner lexer read is carried by the token that opened
it: $token->children holds the inner stream, and
$token->size covers all of it. The outer grammar only
ever mentions <T_OPEN>.
channel(docblocks) hides a token from the parser but
keeps it in the stream, so a documentation generator can work off
the same single pass.
the documentation contains complete grammars: math, calculator, json, json5, phpdoc types and go! aop pointcuts browse
"Syntax error at offset 137" is correct and useless at the same time. Four different things can go wrong while a source is being read, and all four of them are printed the same way.
A plain StringSource can only show the snippet itself.
VirtualSource::createFromString('input.txt', $text) costs
nothing and lets every error above say where it came from.
error[UnexpectedTokenException]: Syntax error, unexpected "and" (T_AND) filter.txt:1:24 price < 100 and brand in and ("acme") ^^^ = the reported position is the furthest point any rule reached, not the place where the last attempt failed
use Phplrt\Parser\Exception\UnexpectedTokenException; use Phplrt\Source\VirtualSource; try { $parser->parse(VirtualSource::createFromString( 'filter.txt', $input, )); } catch (UnexpectedTokenException $e) { echo $e->getMessage(); // just the message echo $e; // the whole block above $e->token->name; // T_AND $e->token->offset; // 23 }
warning: Unrecognized input, skipped filter.txt:1:13 price < 100 § and stock > 0 ^ = the lexer does not stop on unknown input. "§" arrives on the Unknown channel, so a single pass can report a whole file instead of only its first problem
use Phplrt\Contracts\Lexer\Channel; use Phplrt\Exception\ErrorPrinter; use Phplrt\Exception\Printer\Level; foreach ($lexer->lex($source) as $token) { if ($token->channel !== Channel::Unknown) { continue; } echo new ErrorPrinter() ->print($source, $token->offset, $token->size) ->withMessage('Unrecognized input, skipped') ->withLevel(Level::Warning) ->withLinesAround(0); }
error[UnknownFieldException]: Field "brnd" does not exist filter.txt:1:17 price < 100 and brnd = "acme" ^^^^ = did you mean "brand"?
final class FieldException extends \RuntimeException { public function __construct( string $message, public readonly ReadableInterface $source, public readonly int $offset, public readonly int $length = 0, ) { parent::__construct($message); } public function __toString(): string { return (string) new ErrorPrinter() ->print($this->source, $this->offset, $this->length) ->withMessage($this->getMessage()) ->withClass(static::class); } }
// "price < 100 and" - someone is still typing PartialResult read a fragment and stopped value Every(Cond(price, <, 100)) token "and" at offset 12 error - the exception parse() would have thrown filter.txt:1:13 price < 100 and ^^^
use Phplrt\Parser\Analysis\Mode; use Phplrt\Parser\Analysis\Result\PartialResult; use Phplrt\Parser\Analysis\Result\SuccessfulResult; $result = $parser->analyze($source); // SuccessfulResult · PartialResult · FailureResult $result->value; // what it reduced to // a source read in full has nothing more to say if ($result instanceof PartialResult) { $result->token->offset; // where it stopped $result->error; // what stood in the way } // greedy rules separate "not finished yet" from // "written wrong", which is what a search box needs $check = $parser->analyze($typed, Mode::SyntaxCheck); if ($check instanceof SuccessfulResult && !$check instanceof PartialResult) { $button->enable(); }
One renderer with three levels, and colours that adapt to the output
stream. Use withLinesAround(0) when a linter has forty
errors to print at once.
PrinterInterface receives the captured lines and returns a
string, so you can output SARIF, LSP diagnostics, GitHub annotations, or
a single line per error for a CI log.
SourceExceptionInterface, LexerExceptionInterface
and ParserExceptionInterface, so you can catch errors as
broadly or as narrowly as you need.
the three interfaces ship with phplrt/runtime · the printer is phplrt/exception, installed separately
printing errors
On inputs under a kilobyte every parser here reads a type expression in
about a millisecond, and the differences between them are noise. The gap
opens up on larger inputs: phplrt takes ×8.8 longer for every ×10 more
input, which is about as close to linear as parsing gets, while
phpstan/phpdoc-parser takes ×78 longer. On a quarter of a
megabyte that is 24 seconds against 194 ms.
| tool | 25 B | 250 B | 1 KB | 10 KB | 100 KB | 250 KB | per ×10 of input |
|---|---|---|---|---|---|---|---|
| phplrt 4.x | 0.81 | 1.15 | 1.83 | 8.92 | 78.1 | 194 | ×8.8 |
| phplrt 3.x | 0.85 | 1.74 | 4.55 | 35.1 | 335 | 901 | ×9.6 |
| hoa/compiler | 0.85 | 2.79 | 9.99 | 87.1 | 877 | 2 353 | ×10.1 |
| vimeo/psalm | 0.68 | 1.29 | 2.28 | 16.9 | 722 | 5 154 | ×43 |
| phpstan/phpdoc-parser | 0.70 | 1.02 | 2.15 | 47.9 | 3 725 | 24 177 | ×78 |
| tool | 25 B | 250 B | 1 KB | 10 KB | 100 KB | 250 KB | × ours at 250 KB |
|---|---|---|---|---|---|---|---|
| phplrt 4.x | 0.81 | 0.83 | 0.92 | 1.81 | 11.9 | 28.6 | 1.0× |
| vimeo/psalm | 1.57 | 1.61 | 1.79 | 3.63 | 21.0 | 49.2 | 1.7× |
| phpstan/phpdoc-parser | 0.78 | 0.81 | 1.01 | 2.86 | 20.4 | 55.2 | 1.9× |
| phplrt 3.x | 0.83 | 0.87 | 1.10 | 3.28 | 24.0 | 60.8 | 2.1× |
| hoa/compiler | 0.95 | 1.31 | 2.64 | 15.7 | 139 | 336 | 11.8× |
(*MARK:n), so the input is scanned once
Almost every public API has changed. Property hooks, asymmetric visibility
and new in initializers are used throughout, which is why so
much of the code looks different. Grammar files, on the other hand, mostly
compile unchanged.
The API is built on language features that only exist from 8.4 onwards, and those same features are what made the lookahead tables and the code generation practical.
A generated parser is an ordinary class again, with the lexer, the token constants and the reducers as methods. Commit it and the compiler stays a development dependency.
No state is carried between calls, so a single parser instance is safe to share and to call from several places at once.
grammar files carry over · constructors do not upgrade guide
The compiler reads your grammar and writes a parser, which is a job for development time. The runtime reads your users' text. Keep the two apart and production ends up carrying very little.
Read the grammar, generate the parser, commit the result. Run it again whenever the grammar changes, which at the beginning is often.
phplrt/compilerreads .pp3 / .pp2, resolves %include, writes PHPphplrt/lexer-buildera lexer described in PHPphplrt/parser-buildera grammar described in PHP$ composer require phplrt/compiler --dev $ vendor/bin/phplrt compile grammar.pp3 src/RouteParser.php \ --namespace='App\Routing' \ --class=RouteParser
The generated parser is plain PHP that refers only to the runtime. No grammar file, no compilation step, nothing to warm up.
phplrt/sourcefiles, strings and streams, each with a namephplrt/lexertext into tokensphplrt/parsertokens into your result· 6 contractsthe lexer, parser and source interfaces the three packages are written against; the source ones are split across two packagesphplrt/exceptionturns offsets into snippets; a separate package, add it if you print errors$ composer require phplrt/runtime
php 8.4+ · pcre · mbstring recommended · mit installation guide
A search filter, a routing table, a docblock type, an in-house format nobody has published a package for. A grammar takes about half an hour to write, and from then on every mistake in the input gets an error that points at the exact character.
composer require phplrt/phplrt
2017 · started as a fork of hoa/compiler 2019 · a library of its own, developed and maintained ever since