phplrt 4.2
4.2 new lexer · analyze() · generated parsers

Write a grammar,
get a parser.

phplrt is a parsing toolkit for PHP: a lexer, a PEG parser, a grammar compiler and an error printer. You describe your format once, in a file that reads much like the language it describes, and you get back tokens, a syntax tree and error messages that point at the exact character.

composer require phplrt/phplrt
sum.pp3 the grammar
// the tokens
%skip  T_WHITESPACE  \s++
%token T_DIGIT       \d++
%token T_PLUS        \+

// the rules, and what to build out of them
Sum -> { return \array_sum($children); }
  : Number() (::T_PLUS:: Number())*
  ;
Sum the same rule as a diagram
Number() ::T_PLUS:: Number() zero or more times
rule token, kept token, discarded
result parse('2 + 3 + 4') → 9
PHP 8.4+
8 components · 6 contracts
backtracking PEG · FIRST sets
MIT
01 · reading

From text to a tree in two steps

The lexer turns characters into tokens. The parser then checks that those tokens appear in an order the grammar allows and builds the result you asked for. There are no other stages in between.

1 · sourcenamed

A string, a file or a stream, wrapped in an object that knows both its content and its own name. The name is what an error message uses to say where the problem is.

FileSource::createFromPathname(
    pathname: __DIR__ . '/example.txt',
);

StringSource::createFromString(
    content: '2 + 2',
);

ResourceSource::createFromResource(
    resource: $resource,
);
2 · lexerone regex, one pass
nameT_NAME ␣Hidden =T_EQ "phplrt"T_STRING ⏎Hidden versionT_NAME =T_EQ 4T_NUMBER

Every token carries id, name, value, offset and size. Offsets are counted in bytes, which is what lets an error printer cut out the right snippet later.

3 · parserbacktracking PEG

Rules are kept in a table rather than compiled into nested function calls, and each rule knows which tokens it may start with. Recognition writes a flat trace of what matched; the reducers run afterwards, once each, from the innermost rule outwards. A branch that fails is cut off.

Config
├── Pair
│   ├── T_NAME   "name"
│   └── T_STRING "phplrt"
└── Pair
    ├── T_NAME   "version"
    └── T_NUMBER "4"
channels

Hidden tokens are labelled, not dropped

Instead of throwing a token away, the lexer puts it on a channel, and it emits every channel except the ones you tell it to skip. By default that is Hidden alone. Give docblocks a channel of their own and they stay in the stream that the parser reads past. Text the lexer does not recognize arrives as an Unknown token instead of an exception, so a linter can report a whole file rather than stopping at the first problem.

ordered choice

Only one way to read the input

Alternatives are tried from top to bottom and the first one that matches wins, so a grammar can never be ambiguous. The catch is that "a" | "ab" will never match ab, so the longer alternative has to come first.

backtracking PEG recognizer · table-driven recursive descent · FIRST-set prediction · deferred tree construction how the parser works

02 · anatomy

A grammar file has only two kinds of line

One kind declares a token, the other declares a rule. Everything else, including states, channels and reducers, is an optional suffix on one of those two lines.

a token%token · one line per word of the language
%token declares a token. %skip does the same, but hides it from the parser string: the state it belongs to. *: means every state T_QUOTE the name, used by you and by error messages. The parser itself compares ids " the expression. No literal spaces: write \x20 -> state(strings), channel(quotes) what happens once the token is matched: hand the reading over to another lexer, and move the token onto a channel of its own
a rulea name, an optional reducer and a body
Field the name. Referred to as Field() -> { return new Node($offset, $children); } the reducer. It receives whatever the body matched and returns whatever you need: a number, a node, an array. Leave it out and the tokens come through unchanged : <T_NAME> a token, kept ::T_COLON:: a token, discarded Value()? ; another rule, optional. There are also *, + and {2,5}, plus &e / !e for lookahead that reads nothing
01

The first pattern wins

The lexer uses the first pattern that matches, not the longest one. Declare ** before * and keywords before the identifier pattern, or if will be read as a name.

02

No left recursion

Expr : Expr() "+" Number() is rejected when the grammar is built, not at runtime. Write it as a repetition instead, the way the diagram in the hero does.

03

Punctuation needs no %token

Writing "=" inside a rule declares a token for it on the spot, and a token declared this way is always discarded. /and|or/ does the same with a regular expression.

03 · grammars

Three example grammars

A grammar is usually shorter than the code you would write to read the same format by hand, and unlike that code it can be drawn as a diagram.

config.pp318 lines
%skip  T_WHITESPACE  \s++
%skip  T_COMMENT     //[^\n]*+

%token T_NUMBER      \d++(?:\.\d++)?
%token T_STRING      "[^"]*+"
%token T_TRUE        true
%token T_FALSE       false
%token T_NAME        [a-zA-Z_][a-zA-Z0-9_]*+

%pragma root Config

Config : Pair()* ;

Pair : <T_NAME> "=" Value() ;

Value
  : <T_NUMBER>
  | <T_STRING>
  | List()
  ;

List : "[" (Value() ("," Value())*)? "]" ;
Valuethe alternation as a diagram
<T_NUMBER> <T_STRING> List()
rule token, kept
reads
name    = "phplrt"
version = 4
tags    = ["php", "peg"]   // comment is hidden

the documentation contains complete grammars: math, calculator, json, json5, phpdoc types and go! aop pointcuts browse

04 · errors

Every error points at the character that caused it

"Syntax error at offset 137" is correct and useless at the same time. Four different things can go wrong while a source is being read, and all four of them are printed the same way.

the one thing to do

Name your sources

A plain StringSource can only show the snippet itself. VirtualSource::createFromString('input.txt', $text) costs nothing and lets every error above say where it came from.

php read.php
error[UnexpectedTokenException]: Syntax error, unexpected "and" (T_AND)
 --> filter.txt:1:24
  |
1 | price < 100 and brand in and ("acme")
  |                          ^^^
  |
  = the reported position is the furthest point any rule
    reached, not the place where the last attempt failed
read.php
use Phplrt\Parser\Exception\UnexpectedTokenException;
use Phplrt\Source\VirtualSource;

try {
    $parser->parse(VirtualSource::createFromString(
        'filter.txt',
        $input,
    ));
} catch (UnexpectedTokenException $e) {
    echo $e->getMessage();  // just the message
    echo $e;                // the whole block above

    $e->token->name;    // T_AND
    $e->token->offset;  // 23
}
levels

Error · Warning · Debug

One renderer with three levels, and colours that adapt to the output stream. Use withLinesAround(0) when a linter has forty errors to print at once.

format

Or a format of your own

PrinterInterface receives the captured lines and returns a string, so you can output SARIF, LSP diagnostics, GitHub annotations, or a single line per error for a CI log.

contracts

One interface per stage

SourceExceptionInterface, LexerExceptionInterface and ParserExceptionInterface, so you can catch errors as broadly or as narrowly as you need.

the three interfaces ship with phplrt/runtime · the printer is phplrt/exception, installed separately printing errors

05 · speed

Ten times the input, ten times the time

On a short type every parser here takes microseconds, and the hand-written ones are ahead: vimeo/psalm reads array<int, User> in 6.2 µs, phpstan/phpdoc-parser in 16, phplrt in 19. From a kilobyte on phplrt is the fastest of the five, and the gap keeps opening: phplrt takes ×9.9 longer for every ×10 more input, which is about as close to linear as parsing gets, while phpstan/phpdoc-parser takes ×37 longer. On 64 KB that is 33 ms against 425 ms.

time against input size phpdoc type expressions, 16 B → 64 KB both axes logarithmic · a straight line means linear growth
hoa · 405 ms psalm · 208 ms phplrt 3.x · 178 ms phpstan · 425 ms phplrt 4.x · 33.0 ms 10 µs 100 µs 1 ms 10 ms 100 ms 1 s 16 B 64 B 256 B 1 KB 4 KB 16 KB 64 KB
phplrt 4.x phplrt 3.x phpstan/phpdoc-parser vimeo/psalm hoa/compiler ×10 per ×10 · linear
method phpbench, mode of 5 iterations × 20 revs after 5 warm-ups · php 8.5 · opcache on, jit on · every parser builds the same AST · parsers generated from grammars/type.abnf · phplrt/benchmarks
the same numbers, read offtime per parse · lower is better
tool 16 B 64 B 256 B 1 KB 4 KB 16 KB 64 KB per ×10 of input
phplrt 4.x 19.4 µs30.8 µs147 µs527 µs2.07 ms8.15 ms33.0 ms×9.9
phplrt 3.x 110 µs149 µs731 µs2.74 ms10.9 ms42.3 ms178 ms×10.1
hoa/compiler 186 µs286 µs1.45 ms5.51 ms22.8 ms94.6 ms405 ms×10.8
vimeo/psalm 6.2 µs20.5 µs170 µs587 µs2.76 ms17.9 ms208 ms×26
phpstan/phpdoc-parser 16.3 µs22.1 µs141 µs611 µs3.65 ms32.9 ms425 ms×37
read it this way the last column is the growth from 1 KB to 64 KB, brought to a step of ×10 · ×10 means linear growth, and only phpstan and psalm are far from it
memory against input size the same seven inputs both axes logarithmic · lower is better
phplrt 3.x · 14 579 kb phplrt 4.x · 5 439 kb 10 kb 100 kb 1 000 kb 10 000 kb 100 000 kb 16 B 64 B 256 B 1 KB 4 KB 16 KB 64 KB
phplrt 4.x phplrt 3.x phpstan/phpdoc-parser vimeo/psalm hoa/compiler
method peak memory of one parse over the process right before it, after a warm-up parse · php 8.5 · opcache on, jit on · phplrt/benchmarks
the same numbers, read offkb per parse · lower is better
tool 16 B 64 B 256 B 1 KB 4 KB 16 KB 64 KB × ours at 64 KB
phplrt 4.x 2.55.625.51023661 3965 4391.0×
phpstan/phpdoc-parser 8.310.039.61284561 7206 6661.2×
hoa/compiler 4.47.836.31293731 9577 6011.4×
vimeo/psalm 4.911.754.01565472 0527 8931.5×
phplrt 3.x 20.520.169.12589903 75014 5792.7×
read it this way what the parse itself allocates, the AST it returns included · the code of the parser is not counted
how much work do you need?4.x · share of a full parse
Syntax Checkis it valid, and how far does it get? no reducers run
66%
Tolerant Modethe same work, but problems are reported instead of thrown
100%
Parsebuilds the full result, throws on a mistake
100%
⚠
These numbers are relative. They are not absolute performance figures; they only show how the three modes compare with each other. Use them to see which mode is cheaper and which is more expensive, not to predict what any of them will cost in your own application.
1 regex per lexer state the alternatives are marked with (*MARK:n), so the input is scanned once
9 packages in the runtime the lexer, the parser and the source reader, plus the six contract packages they are written against. Take the lexer without the parser, or the contracts on their own. Nothing outside phplrt is required
1× grammar compilation, ever compile the grammar once, commit the generated PHP, and stop paying for it on every request
06 · new in 4

A rewritten API, the same grammar files

Almost every public API has changed. Property hooks, asymmetric visibility and new in initializers are used throughout, which is why so much of the code looks different. Grammar files, on the other hand, mostly compile unchanged.

3.x 4.x
$token->getName(), $token->getBytes()
→
$token->name, $token->size
$source->getContents()
→
$source->content
new File($path), new Source($text)
→
FileSource, StringSource,
ResourceSource, VirtualSource
$token->getName() === 'T_DIGIT'
→
$token->id === Parser::T_DIGIT
new Lexer($tokens, ['T_WHITESPACE'])
→
$builder->addPattern('\s++')
  ->hide()
new Parser($lexer, $rules, [...$handlers])
→
new AppParser() // generated
phplrt/buffer · phplrt/visitor · ast-contracts
→
removed. The buffer is now internal to the parser, and 4.x no longer prescribes a node shape
requires

PHP 8.4

The API is built on language features that only exist from 8.4 onwards, and those same features are what made the lookahead tables and the code generation practical.

back

Code generation

A generated parser is an ordinary class again, with the lexer, the token constants and the reducers as methods. Commit it and the compiler stays a development dependency.

safer

Nothing is kept on the parser

No state is carried between calls, so a single parser instance is safe to share and to call from several places at once.

grammar files carry over · constructors do not upgrade guide

07 · install

Half of phplrt never reaches production

The compiler reads your grammar and writes a parser, which is a job for development time. The runtime reads your users' text. Keep the two apart and production ends up carrying very little.

while developing

Describe and compile

Read the grammar, generate the parser, commit the result. Run it again whenever the grammar changes, which at the beginning is often.

phplrt/compilerreads .pp3 / .pp2, resolves %include, writes PHP
phplrt/lexer-buildera lexer described in PHP
phplrt/parser-buildera grammar described in PHP
$ composer require phplrt/compiler --dev

$ vendor/bin/phplrt compile grammar.pp3 src/RouteParser.php \
      --namespace='App\Routing' \
      --class=RouteParser
what ships

Read and report

The generated parser is plain PHP that refers only to the runtime. No grammar file, no compilation step, nothing to warm up.

phplrt/sourcefiles, strings and streams, each with a name
phplrt/lexertext into tokens
phplrt/parsertokens into your result
· 6 contractsthe lexer, parser and source interfaces the three packages are written against; the source ones are split across two packages
phplrt/exceptionturns offsets into snippets; a separate package, add it if you print errors
$ composer require phplrt/runtime

php 8.4+ · pcre · mbstring recommended · mit installation guide

Write the rules down once

A search filter, a routing table, a docblock type, an in-house format nobody has published a package for. A grammar takes about half an hour to write, and from then on every mistake in the input gets an error that points at the exact character.

composer require phplrt/phplrt

2017 · started as a fork of hoa/compiler 2019 · a library of its own, developed and maintained ever since