phplrt 4.0
4.0 new lexer · analyze() · generated parsers

Write a grammar,
get a parser.

phplrt is a parsing toolkit for PHP: a lexer, a PEG parser, a grammar compiler and an error printer. You describe your format once, in a file that reads much like the language it describes, and you get back tokens, a syntax tree and error messages that point at the exact character.

composer require phplrt/phplrt
sum.pp3 the grammar
// the tokens
%skip  T_WHITESPACE  \s++
%token T_DIGIT       \d++
%token T_PLUS        \+

// the rules, and what to build out of them
Sum -> { return \array_sum($children); }
  : Number() (::T_PLUS:: Number())*
  ;
Sum the same rule as a diagram
Number() ::T_PLUS:: Number() zero or more times
rule token, kept token, discarded
result parse('2 + 3 + 4')9
PHP 8.4+
8 components · 6 contracts
backtracking PEG · FIRST sets
MIT
01 · reading

From text to a tree in two steps

The lexer turns characters into tokens. The parser then checks that those tokens appear in an order the grammar allows and builds the result you asked for. There are no other stages in between.

1 · sourcenamed

A string, a file or a stream, wrapped in an object that knows both its content and its own name. The name is what an error message uses to say where the problem is.

FileSource::createFromPathname(
    pathname: __DIR__ . '/example.txt',
);

StringSource::createFromString(
    content: '2 + 2',
);

ResourceSource::createFromResource(
    resource: $resource,
);
2 · lexerone regex, one pass
nameT_NAME Hidden =T_EQ "phplrt"T_STRING Hidden versionT_NAME =T_EQ 4T_NUMBER

Every token carries id, name, value, offset and size. Offsets are counted in bytes, which is what lets an error printer cut out the right snippet later.

3 · parserbacktracking PEG

Rules are kept in a table rather than compiled into nested function calls, and each rule knows which tokens it may start with. Recognition writes a flat trace of what matched; the reducers run afterwards, once each, from the innermost rule outwards. A branch that fails is cut off.

Config
├── Pair
│   ├── T_NAME   "name"
│   └── T_STRING "phplrt"
└── Pair
    ├── T_NAME   "version"
    └── T_NUMBER "4"
channels

Hidden tokens are labelled, not dropped

Instead of throwing a token away, the lexer puts it on a channel, and it emits every channel except the ones you tell it to skip. By default that is Hidden alone. Give docblocks a channel of their own and they stay in the stream that the parser reads past. Text the lexer does not recognize arrives as an Unknown token instead of an exception, so a linter can report a whole file rather than stopping at the first problem.

ordered choice

Only one way to read the input

Alternatives are tried from top to bottom and the first one that matches wins, so a grammar can never be ambiguous. The catch is that "a" | "ab" will never match ab, so the longer alternative has to come first.

backtracking PEG recognizer · table-driven recursive descent · FIRST-set prediction · deferred tree construction how the parser works

02 · anatomy

A grammar file has only two kinds of line

One kind declares a token, the other declares a rule. Everything else, including states, channels and reducers, is an optional suffix on one of those two lines.

a token%token · one line per word of the language
%token declares a token. %skip does the same, but hides it from the parser string: the state it belongs to. *: means every state T_QUOTE the name, used by you and by error messages. The parser itself compares ids " the expression. No literal spaces: write \x20 -> state(strings), channel(quotes) what happens once the token is matched: hand the reading over to another lexer, and move the token onto a channel of its own
a rulea name, an optional reducer and a body
Field the name. Referred to as Field() -> { return new Node($offset, $children); } the reducer. It receives whatever the body matched and returns whatever you need: a number, a node, an array. Leave it out and the tokens come through unchanged : <T_NAME> a token, kept ::T_COLON:: a token, discarded Value()? ; another rule, optional. There are also *, + and {2,5}, plus &e / !e for lookahead that reads nothing
01

The first pattern wins

The lexer uses the first pattern that matches, not the longest one. Declare ** before * and keywords before the identifier pattern, or if will be read as a name.

02

No left recursion

Expr : Expr() "+" Number() is rejected when the grammar is built, not at runtime. Write it as a repetition instead, the way the diagram in the hero does.

03

Punctuation needs no %token

Writing "=" inside a rule declares a token for it on the spot, and a token declared this way is always discarded. /and|or/ does the same with a regular expression.

03 · grammars

Three example grammars

A grammar is usually shorter than the code you would write to read the same format by hand, and unlike that code it can be drawn as a diagram.

config.pp318 lines
%skip  T_WHITESPACE  \s++
%skip  T_COMMENT     //[^\n]*+

%token T_NUMBER      \d++(?:\.\d++)?
%token T_STRING      "[^"]*+"
%token T_TRUE        true
%token T_FALSE       false
%token T_NAME        [a-zA-Z_][a-zA-Z0-9_]*+

%pragma root Config

Config : Pair()* ;

Pair : <T_NAME> "=" Value() ;

Value
  : <T_NUMBER>
  | <T_STRING>
  | List()
  ;

List : "[" (Value() ("," Value())*)? "]" ;
Valuethe alternation as a diagram
<T_NUMBER> <T_STRING> List()
rule token, kept
reads
name    = "phplrt"
version = 4
tags    = ["php", "peg"]   // comment is hidden

the documentation contains complete grammars: math, calculator, json, json5, phpdoc types and go! aop pointcuts browse

04 · errors

Every error points at the character that caused it

"Syntax error at offset 137" is correct and useless at the same time. Four different things can go wrong while a source is being read, and all four of them are printed the same way.

the one thing to do

Name your sources

A plain StringSource can only show the snippet itself. VirtualSource::createFromString('input.txt', $text) costs nothing and lets every error above say where it came from.

php read.php
error[UnexpectedTokenException]: Syntax error, unexpected "and" (T_AND)
 --> filter.txt:1:24
  |
1 | price < 100 and brand in and ("acme")
  |                          ^^^
  |
  = the reported position is the furthest point any rule
    reached, not the place where the last attempt failed
read.php
use Phplrt\Parser\Exception\UnexpectedTokenException;
use Phplrt\Source\VirtualSource;

try {
    $parser->parse(VirtualSource::createFromString(
        'filter.txt',
        $input,
    ));
} catch (UnexpectedTokenException $e) {
    echo $e->getMessage();  // just the message
    echo $e;                // the whole block above

    $e->token->name;    // T_AND
    $e->token->offset;  // 23
}
levels

Error · Warning · Debug

One renderer with three levels, and colours that adapt to the output stream. Use withLinesAround(0) when a linter has forty errors to print at once.

format

Or a format of your own

PrinterInterface receives the captured lines and returns a string, so you can output SARIF, LSP diagnostics, GitHub annotations, or a single line per error for a CI log.

contracts

One interface per stage

SourceExceptionInterface, LexerExceptionInterface and ParserExceptionInterface, so you can catch errors as broadly or as narrowly as you need.

the three interfaces ship with phplrt/runtime · the printer is phplrt/exception, installed separately printing errors

05 · speed

Ten times the input, ten times the time

On inputs under a kilobyte every parser here reads a type expression in about a millisecond, and the differences between them are noise. The gap opens up on larger inputs: phplrt takes ×8.8 longer for every ×10 more input, which is about as close to linear as parsing gets, while phpstan/phpdoc-parser takes ×78 longer. On a quarter of a megabyte that is 24 seconds against 194 ms.

time against input size phpdoc type expressions, 25 B → 250 KB both axes logarithmic · a straight line means linear growth
hoa · 2.35 s psalm · 5.15 s phplrt 3.x · 901 ms phpstan · 24.2 s phplrt 4.x · 194 ms 1 ms 10 ms 100 ms 1 s 10 s 25 B 250 B 1 KB 10 KB 100 KB 250 KB
phplrt 4.x phplrt 3.x phpstan/phpdoc-parser vimeo/psalm hoa/compiler ×10 per ×10 · linear
method phpbench, mode of the run · php 8.5 · opcache on, jit on · parsers generated from grammars/type.abnf · phplrt/benchmarks
the same numbers, read offms per parse · lower is better
tool 25 B 250 B 1 KB 10 KB 100 KB 250 KB per ×10 of input
phplrt 4.x 0.811.151.838.9278.1194×8.8
phplrt 3.x 0.851.744.5535.1335901×9.6
hoa/compiler 0.852.799.9987.18772 353×10.1
vimeo/psalm 0.681.292.2816.97225 154×43
phpstan/phpdoc-parser 0.701.022.1547.93 72524 177×78
read it this way the last column divides the 100 KB cell by the 10 KB cell · ×10 means linear growth, and only phpstan and psalm are far from it
peak memory against input size the same five inputs both axes logarithmic · lower is better
hoa · 336 mb phplrt 4.x · 28.6 mb 1 mb 10 mb 100 mb 25 B 250 B 1 KB 10 KB 100 KB 250 KB
phplrt 4.x phplrt 3.x phpstan/phpdoc-parser vimeo/psalm hoa/compiler
method phpbench, peak memory of one parse · php 8.5 · opcache on, jit on · phplrt/benchmarks
the same numbers, read offmb per parse · lower is better
tool 25 B 250 B 1 KB 10 KB 100 KB 250 KB × ours at 250 KB
phplrt 4.x 0.810.830.921.8111.928.61.0×
vimeo/psalm 1.571.611.793.6321.049.21.7×
phpstan/phpdoc-parser 0.780.811.012.8620.455.21.9×
phplrt 3.x 0.830.871.103.2824.060.82.1×
hoa/compiler 0.951.312.6415.713933611.8×
read it this way below a kilobyte this is the runtime's own footprint rather than the parse · rstdev across every run above: ±0.52…3.32%
how much work do you need?4.x · share of a full parse
Syntax Checkis it valid, and how far does it get? no reducers run
54%
Tolerant Modethe same work, but problems are reported instead of thrown
99%
Parsebuilds the full result, throws on a mistake
100%
These numbers are relative. They are not absolute performance figures; they only show how the three modes compare with each other. Use them to see which mode is cheaper and which is more expensive, not to predict what any of them will cost in your own application.
1 regex per lexer state the alternatives are marked with (*MARK:n), so the input is scanned once
9 packages in the runtime the lexer, the parser and the source reader, plus the six contract packages they are written against. Take the lexer without the parser, or the contracts on their own. Nothing outside phplrt is required
grammar compilation, ever compile the grammar once, commit the generated PHP, and stop paying for it on every request
06 · new in 4

A rewritten API, the same grammar files

Almost every public API has changed. Property hooks, asymmetric visibility and new in initializers are used throughout, which is why so much of the code looks different. Grammar files, on the other hand, mostly compile unchanged.

3.x 4.0
$token->getName(), $token->getBytes()
$token->name, $token->size
$source->getContents()
$source->content
new File($path), new Source($text)
FileSource, StringSource,
ResourceSource, VirtualSource
$token->getName() === 'T_DIGIT'
$token->id === Parser::T_DIGIT
new Lexer($tokens, ['T_WHITESPACE'])
$builder->addPattern('\s++')
  ->hide()
new Parser($lexer, $rules, [...$handlers])
new AppParser() // generated
phplrt/buffer · phplrt/visitor · ast-contracts
removed. The buffer is now internal to the parser, and 4.x no longer prescribes a node shape
requires

PHP 8.4

The API is built on language features that only exist from 8.4 onwards, and those same features are what made the lookahead tables and the code generation practical.

back

Code generation

A generated parser is an ordinary class again, with the lexer, the token constants and the reducers as methods. Commit it and the compiler stays a development dependency.

safer

Nothing is kept on the parser

No state is carried between calls, so a single parser instance is safe to share and to call from several places at once.

grammar files carry over · constructors do not upgrade guide

07 · install

Half of phplrt never reaches production

The compiler reads your grammar and writes a parser, which is a job for development time. The runtime reads your users' text. Keep the two apart and production ends up carrying very little.

while developing

Describe and compile

Read the grammar, generate the parser, commit the result. Run it again whenever the grammar changes, which at the beginning is often.

phplrt/compilerreads .pp3 / .pp2, resolves %include, writes PHP
phplrt/lexer-buildera lexer described in PHP
phplrt/parser-buildera grammar described in PHP
$ composer require phplrt/compiler --dev

$ vendor/bin/phplrt compile grammar.pp3 src/RouteParser.php \
      --namespace='App\Routing' \
      --class=RouteParser
what ships

Read and report

The generated parser is plain PHP that refers only to the runtime. No grammar file, no compilation step, nothing to warm up.

phplrt/sourcefiles, strings and streams, each with a name
phplrt/lexertext into tokens
phplrt/parsertokens into your result
· 6 contractsthe lexer, parser and source interfaces the three packages are written against; the source ones are split across two packages
phplrt/exceptionturns offsets into snippets; a separate package, add it if you print errors
$ composer require phplrt/runtime

php 8.4+ · pcre · mbstring recommended · mit installation guide

Write the rules down once

A search filter, a routing table, a docblock type, an in-house format nobody has published a package for. A grammar takes about half an hour to write, and from then on every mistake in the input gets an error that points at the exact character.

composer require phplrt/phplrt

2017 · started as a fork of hoa/compiler 2019 · a library of its own, developed and maintained ever since