phplrt 4.0
4.0 new lexer · analyze() · generated parsers

Read anything
that has rules in it.

A lexer, a PEG parser, a grammar compiler and an error printer for PHP. Describe your format once — in a file that reads like the language it describes — and get tokens, a tree, and errors that point at the character.

composer require phplrt/phplrt
sum.pp3 the grammar
// the words
%skip  T_WHITESPACE  \s++
%token T_DIGIT       \d++
%token T_PLUS        \+

// the sentences, and what to build out of them
Sum -> { return \array_sum($children); }
  : Number() (::T_PLUS:: Number())*
  ;
Sum the same rule, drawn
Number() ::T_PLUS:: Number() zero or more times
rule token, kept token, discarded
result parse('2 + 3 + 4')9
PHP 8.4+
6 loosely coupled packages
backtracking PEG · FIRST sets
MIT
01 · reading

Two halves: one cuts, one arranges

The lexer turns characters into tokens. The parser checks that those tokens appear in an order the grammar allows, and builds whatever you asked for. Nothing else happens in between.

1 · sourcenamed

A string, a file or a stream — wrapped in an object that knows its content and what to call itself when an error has to say where.

new File(__DIR__ . '/example.txt');

new VirtualFile('config.txt', $text);

new Source('2 + 2');

new Stream($resource);
2 · lexerone regex, one pass
nameT_NAME Hidden =T_EQ "phplrt"T_STRING Hidden versionT_NAME =T_EQ 4T_NUMBER

Every token carries id, name, value, offset and size — offsets in bytes, which is what makes snippets possible later.

3 · parserbacktracking PEG

Rules live in a table, not in a call graph, and each one knows the tokens it may start with. Recognition writes a flat trace; the reducers run once, afterwards, bottom-up. A branch that fails is simply truncated.

Config
├── Pair
│   ├── T_NAME   "name"
│   └── T_STRING "phplrt"
└── Pair
    ├── T_NAME   "version"
    └── T_NUMBER "4"
channels

Nothing is thrown away

Whitespace and comments are hidden, not skipped: the parser does not see them, a documentation generator can. Unrecognized text becomes an Unknown token instead of an exception, so a linter reports a whole file rather than its first problem.

ordered choice

There is only one reading

Alternatives are tried top to bottom and the first match wins, so a grammar is never ambiguous. The flip side: "a" | "ab" never reads ab — put the longer one first.

backtracking PEG recognizer · table-driven recursive descent · FIRST-set prediction · deferred tree construction how the parser works

02 · anatomy

Two kinds of line, and that is the whole syntax

A grammar file says what the words are and what order they may come in. Everything else — states, channels, reducers — is an optional suffix on one of these two lines.

a word%token · one line, read by a lexer of its own
%token declares a token. %skip is the same one, hidden from the parser string: the state it belongs to. *: means every state T_QUOTE the name — for you and for error messages. The parser compares ids " the expression. No literal spaces: write \x20 -> state(strings), channel(quotes) what the token does besides being read: hand the reading over to another lexer, and put itself on a channel of its own
a sentencea rule · name, an optional reducer, a body
Field the name. Referred to as Field() -> { return new Node($offset, $children); } the reducer. Gets whatever the body matched and returns whatever you want — a number, a node, an array. Omit it and the tokens come through as they are : <T_NAME> a token, kept ::T_COLON:: a token, discarded Value()? ; another rule, optional. Also *, +, {2,5}, and &e / !e to look ahead without reading
01

First match, not longest

The lexer takes the first pattern that matches. Declare ** before *, and keywords before the identifier pattern — or if becomes a name.

02

No left recursion

Expr : Expr() "+" Number() is refused while building, not at runtime. Write it as a repetition — which is what the diagram in the hero shows anyway.

03

Punctuation declares itself

"=" inside a rule declares its own token, and such a token is always discarded. /and|or/ does the same with a regular expression.

03 · grammars

Three languages, none of them long

A grammar is normally shorter than the code you would write to read the same thing by hand — and unlike that code, it can be drawn.

config.pp318 lines
%skip  T_WHITESPACE  \s++
%skip  T_COMMENT     //[^\n]*+

%token T_NUMBER      \d++(?:\.\d++)?
%token T_STRING      "[^"]*+"
%token T_TRUE        true
%token T_FALSE       false
%token T_NAME        [a-zA-Z_][a-zA-Z0-9_]*+

%pragma root Config

Config : Pair()* ;

Pair : <T_NAME> "=" Value() ;

Value
  : <T_NUMBER>
  | <T_STRING>
  | List()
  ;

List : "[" (Value() ("," Value())*)? "]" ;
Valuean alternation, drawn
<T_NUMBER> <T_STRING> List()
rule token, kept
reads
name    = "phplrt"
version = 4
tags    = ["php", "peg"]   // comment is hidden

the docs ship complete grammars: math, calculator, json, json5, phpdoc types, go! aop pointcuts browse

04 · errors

Every stage points at the character it broke on

“Syntax error at offset 137” is technically correct and practically useless. Four things can go wrong while reading a source — all four print the same way.

the one thing to do

Name your sources

A bare Source can only show the snippet. new VirtualFile('input.txt', $text) costs nothing and lets every error above say where.

php read.php
error[UnexpectedTokenException]: Syntax error, unexpected "and" (T_AND)
 --> filter.txt:1:24
  |
1 | price < 100 and brand in and ("acme")
  |                          ^^^
  |
  = the position reported is the furthest one any rule
    reached, not where the last attempt gave up
read.php
use Phplrt\Parser\Exception\UnexpectedTokenException;
use Phplrt\Source\VirtualFile;

try {
    $parser->parse(new VirtualFile('filter.txt', $input));
} catch (UnexpectedTokenException $e) {
    echo $e->getMessage();  // the sentence
    echo $e;                // …and the block above

    $e->token->name;    // T_AND
    $e->token->offset;  // 23
}
levels

Error · Warning · Debug

One renderer, three levels, colours that follow the output stream. withLinesAround(0) when a linter has forty of them.

format

Or a format of your own

PrinterInterface takes the captured lines and returns a string: SARIF, LSP diagnostics, GitHub annotations, one line per error for a CI log.

contracts

One interface per stage

SourceExceptionInterface, LexerExceptionInterface, ParserExceptionInterface — catch as coarsely or as finely as the situation deserves.

the three interfaces ship with phplrt/runtime · the printer is phplrt/exception, installed separately printing errors

05 · speed

Ten times the input, ten times the time

Under a kilobyte every one of these reads a type expression in about a millisecond and the difference between them is noise. It is the hundredth kilobyte that separates them: phplrt costs ×8.8 for every ×10 of input, which is as close to linear as reading gets. phpstan/phpdoc-parser costs ×78 — and at a quarter of a megabyte that is 24 seconds against 194 ms.

time against input size phpdoc type expressions, 25 B → 250 KB both axes logarithmic — a straight line is linear growth
hoa · 2.35 s psalm · 5.15 s phplrt 3.x · 901 ms phpstan · 24.2 s phplrt 4.x · 194 ms 1 ms 10 ms 100 ms 1 s 10 s 25 B 250 B 1 KB 10 KB 100 KB 250 KB
phplrt 4.x phplrt 3.x phpstan/phpdoc-parser vimeo/psalm hoa/compiler ×10 per ×10 — linear
method phpbench, mode of the run · php 8.5 · opcache on, jit on · parsers generated from grammars/type.abnf · phplrt/benchmarks
the same numbers, read offms per parse · lower is better
tool 25 B 250 B 1 KB 10 KB 100 KB 250 KB per ×10 of input
phplrt 4.x 0.811.151.838.9278.1194×8.8
phplrt 3.x 0.851.744.5535.1335901×9.6
hoa/compiler 0.852.799.9987.18772 353×10.1
vimeo/psalm 0.681.292.2816.97225 154×43
phpstan/phpdoc-parser 0.701.022.1547.93 72524 177×78
read it this way the last column is the 100 KB cell over the 10 KB cell — ×10 is linear, and only phpstan and psalm are far from it
peak memory against input size the same five inputs both axes logarithmic · lower is better
hoa · 336 mb phplrt 4.x · 28.6 mb 1 mb 10 mb 100 mb 25 B 250 B 1 KB 10 KB 100 KB 250 KB
phplrt 4.x phplrt 3.x phpstan/phpdoc-parser vimeo/psalm hoa/compiler
method phpbench, peak memory of one parse · php 8.5 · opcache on, jit on · phplrt/benchmarks
the same numbers, read offmb per parse · lower is better
tool 25 B 250 B 1 KB 10 KB 100 KB 250 KB × ours at 250 KB
phplrt 4.x 0.810.830.921.8111.928.61.0×
vimeo/psalm 1.571.611.793.6321.049.21.7×
phpstan/phpdoc-parser 0.780.811.012.8620.455.21.9×
phplrt 3.x 0.830.871.103.2824.060.82.1×
hoa/compiler 0.951.312.6415.713933611.8×
read it this way below a kilobyte you are looking at the runtime's own footprint, not the parse · rstdev across every run above ±0.52…3.32%
how much work do you need?4.x · share of a full parse
Syntax Checkvalid? and how far — no reducers run
54%
Tolerant Modethe same work, reported instead of thrown
99%
Parse builds everything, throws on a mistake
100%
Relative, not absolute. These figures are not real-world performance in absolute units — only the order and the relative difference between the modes. They tell you which mode is faster and which is slower, not the exact numbers any of them will cost you.
1 regex per lexer state alternatives marked with (*MARK:n), one pass over the input
6 loosely coupled packages three components and the three interface packages they are written against — take the lexer without the parser, or the contracts on their own. Nothing outside phplrt
grammar compilation, ever compile once, commit the PHP, stop paying for it per request
06 · new in 4

A rewrite that kept your grammar files

Almost every public API changed — property hooks, asymmetric visibility and new in initializers are used throughout, which is why so much of it looks different. Grammar files mostly compile unchanged.

3.x 4.0
$token->getName(), $token->getBytes()
$token->name, $token->size
$source->getContents()
$source->content
$token->getName() === 'T_DIGIT'
$token->id === Parser::T_DIGIT
new Lexer($tokens, ['T_WHITESPACE'])
$builder->addPattern('\s++')
  ->hide()
new Parser($lexer, $rules, [...$handlers])
new AppParser() // generated
phplrt/buffer · phplrt/position · phplrt/visitor
removed — 4.x prescribes no node shape
requires

PHP 8.4

The API is built on features that only exist there, which is also what made the lookahead tables and the code generation practical.

back

Code generation

A generated parser is a real class again — the lexer, the token constants and the reducers as methods. Commit it and the compiler stays a dev dependency.

safer

Nothing is kept on the parser

No execution context between calls, so one parser is safe to share and to call from several places at once.

grammar files carry over · the constructors do not upgrade guide

07 · install

One half never reaches production

The compiler reads your grammar and writes a parser — a development job. The runtime reads your users' text. Keep them apart and production carries very little.

while developing

Describe and compile

Read the grammar, generate the parser, commit the result. Re-run it when the grammar changes — which, at the start, is every five minutes.

phplrt/compilerreads .pp3 / .pp2, resolves %include, writes PHP
phplrt/lexer-buildera lexer described in PHP
phplrt/parser-buildera grammar described in PHP
$ composer require phplrt/compiler --dev

$ vendor/bin/phplrt compile grammar.pp3 src/RouteParser.php \
      --namespace='App\Routing' \
      --class=RouteParser
what ships

Read and report

The generated parser is plain PHP referring to the runtime — no grammar file, no compilation, nothing to warm up.

phplrt/sourcefiles, strings, streams — with names
phplrt/lexertext into tokens
phplrt/parsertokens into your result
· 3 contractsthe source, lexer and parser interfaces the three are written against
phplrt/exceptionoffsets into snippets — a separate package, add it if you print errors
$ composer require phplrt/runtime

php 8.4+ · pcre · mbstring recommended · mit installation guide

If it has rules, write them down

A search filter, a routing table, a docblock type, an in-house format nobody wrote a package for. Half an hour to a grammar, and every mistake in it gets an error that points at the character.

composer require phplrt/phplrt