Upgrade Guide

Upgrading To 4.0 From 3.x

Version 4.0 is a rewrite. Almost every public API changed, so this is a porting guide rather than a list of renames. The good news: the grammar files mostly survive, and that is where the bulk of the work usually lives.

PHP 8.4 Required

Likelihood Of Impact: High

Phplrt 4.0 requires PHP 8.4. The API uses property hooks, asymmetric visibility and new in initializers throughout - which is also why so much of it looks different.

Getters Became Properties

Likelihood Of Impact: High

Everything that was getX() is now a property. This is the single most common change you will hit.

 1// 3.x
 2$token->getName();
 3$token->getOffset();
 4$token->getValue();
 5$token->getBytes();
 6
 7$source->getContents();
 8$source->getPathname();
 9
10// 4.x
11$token->name;
12$token->offset;
13$token->value;
14$token->size;
15
16$source->content;
17$source->pathname;

Tokens Are Identified By Number

Likelihood Of Impact: High

In 3.x a token was addressed by its name. In 4.x it has an int $id, and the name is optional metadata for error messages.

1// 3.x
2if ($token->getName() === 'T_DIGIT') { /* ... */ }
3
4// 4.x
5if ($token->id === MyParser::T_DIGIT) { /* ... */ }

Generated parsers expose the ids as class constants, so you do not have to track the numbers yourself.

Channels Replaced "Skipped Tokens"

Likelihood Of Impact: Medium

The lexer no longer takes a list of names to skip. Every token now carries a channel, and the lexer reports every channel except the ones it is told to leave out - Hidden alone, unless you say otherwise.

1// 3.x
2$lexer = new Lexer($tokens, ['T_WHITESPACE']);
3
4// 4.x
5$builder->addPattern('\s++', 'T_WHITESPACE')
6    ->hide();

Unlike a skipped name, a channel is a decision you can revisit: the same token description gives you a lexer reporting the whitespace and the comments when you need to look at them, and custom channels let you keep a token in the stream and still tell it apart from the code.

The Lexer Is Built, Not Configured

Likelihood Of Impact: High

Phplrt\Lexer\Lexer no longer takes a map of names and patterns. It takes a single compiled regular expression and a set of tables - which you get from LexerBuilder.

 1// 3.x
 2$lexer = new Lexer([
 3    'T_DIGIT' => '\d+',
 4    'T_PLUS'  => '\+',
 5]);
 6
 7// 4.x
 8use Phplrt\Lexer\Builder\LexerBuilder;
 9
10$builder = new LexerBuilder();
11$builder->addPattern('\d++', 'T_DIGIT');
12$builder->addValue('+', 'T_PLUS');
13
14$lexer = $builder->build()
15    ->toLexer();

The append(), prepend(), prependMany() and skip() methods are gone; addPattern(), addValue() and hide() cover the same ground. Building is also where patterns are validated, so a broken regex is reported with the token that owns it instead of failing at match time.

The Parser Takes Explicit Arguments

Likelihood Of Impact: High

The Parser::CONFIG_* options are gone, replaced by constructor parameters:

 1// 3.x
 2$parser = new Parser($lexer, $grammar, [
 3    Parser::CONFIG_INITIAL_RULE => 'expression',
 4    Parser::CONFIG_AST_BUILDER  => new MyBuilder(),
 5]);
 6
 7// 4.x
 8$parser = new Parser(
 9    lexer: $lexer,
10    grammar: $grammar,
11    initial: 0,
12    reducers: [0 => $callback],
13);

Rules are keyed by integers rather than by name, and they refer to each other by index. In practice you do not write this array by hand - see the parser builder.

Checking And Trailing Tokens Became One Method

Likelihood Of Impact: Medium

3.x answered "is this valid?" with check(), and read a source the grammar does not describe in full through a setting, picking up where it stopped from the parser afterwards. Both are analyze() now, and it returns what it found rather than keeping it:

 1// 3.x
 2$parser->check($source); // true or false
 3
 4$parser = new Parser(..., [Parser::CONFIG_ALLOW_TRAILING_TOKENS => true]);
 5$parser->parse($source);
 6
 7$context = $parser->getLastExecutionContext();
 8$context->buffer->current(); // where the parser stopped
 9
10// 4.x
11use Phplrt\Parser\Analysis\Mode;
12use Phplrt\Parser\Analysis\Result\PartialResult;
13use Phplrt\Parser\Analysis\Result\SuccessfulResult;
14
15$result = $parser->analyze($source, Mode::SyntaxCheck);
16
17$result instanceof SuccessfulResult; // the grammar has read something
18$result instanceof PartialResult;    // ...and there is more to read
19
20$result = $parser->analyze($source);
21
22$result->value; // what has been read reduced to
23
24// A source read in full has nothing more to say, so these belong to the two
25// results that mean it has not been
26$result->token;      // where the parser stopped
27$result->error;      // and the exception it would be rejected with

Two things changed beyond the names. Nothing is kept on the parser between calls, so a parser is safe to share and to call from several places at once. And where the reading stopped is now told for every source, not only for the ones that fail: a valid source is a SuccessfulResult, a source the grammar reads in part is a PartialResult, and one it cannot read at all is a FailureResult.

See Analysing A Source.

BuilderInterface Became Per-Rule Reducers

Likelihood Of Impact: High

The single AST builder with a switch overrule names is gone. Each rule now carries its own reducer.

 1// 3.x
 2class MyBuilder implements BuilderInterface
 3{
 4    public function build(Context $ctx, $children)
 5    {
 6        switch ($ctx->getState()) {
 7            case 'Number': return new NumberNode($children);
 8            case 'Sum':    return new SumNode($children);
 9        }
10
11        return null;
12    }
13}
14
15// 4.x - in the grammar file
16Number -> { return new \NumberNode($offset, $children->value); } 
17    : <T_DIGIT> ;
18Sum    -> { return new \SumNode($children); } 
19    : Number() ::T_PLUS:: Number() ;

1// 4.x - or through the builder
2$number->setReducer(static fn(Context $ctx, mixed $children): NumberNode
3    => new NumberNode($children)
4);

The reducer signature is callable(Context $ctx, mixed $children): mixed, same as before, but $ctx->getState() is now $ctx->rule and holds an integer.

Returning null still means "leave the children alone".

Grammar Rule Classes

Likelihood Of Impact: Medium

The rules moved from Phplrt\Parser\Grammar (3.2) - same namespace, but they are now readonly value objects that reference other rules by integer id, and Lexeme takes a token id instead of a name.

1// 3.x
2new Lexeme('T_DIGIT');
3new Lexeme('T_WHITESPACE', false);
4new Repetition($ruleId, 0, \INF);
5
6// 4.x
7new Lexeme(MyLexer::T_DIGIT);
8new Lexeme(MyLexer::T_WHITESPACE, false);
9new Repetition($ruleId, 0, \INF);

reduce() is gone from the rule classes: matching is done by the parser's internal tracer, and the rules are pure data. This is what made the up-front analysis and code generation possible.

The Buffer Package Is Gone

Likelihood Of Impact: Low

phplrt/buffer was merged into phplrt/parser as an internal detail (Phplrt\Parser\Internal\Buffer). The parser no longer exposes a buffer, and you do not choose one.

The Position and Visitor Packages Are Gone

Likelihood Of Impact: Medium

phplrt/position and phplrt/visitor have been removed.

Positions: the exception component computes lines and columns when it renders an error, which was the main use for it. See Error Reporting.

Visitors: 4.x does not prescribe an AST shape, so it cannot prescribe a way to walk one. Your nodes are your own classes; walk them however suits them.

Sources Are Constructed Directly

Likelihood Of Impact: Medium

The static factory methods on File are gone.

1// 3.x
2File::fromPathname('/app/x.txt');
3File::fromSources('2 + 2');
4
5// 4.x
6new File('/app/x.txt');
7new Source('2 + 2');
8new VirtualFile('x.txt', '2 + 2'); // a string with a name

SourceFactory::createDefault() is there if you need the "figure out what this is" behaviour. Note that it now has a single create() method: the createFromString(), createFromFile() and createFromStream() helpers are gone, and each kind of source is a driver behind that one method instead.

Code Generation Is Back

Likelihood Of Impact: Low

3.0 removed generation of a full PHP class in favour of a config array. 4.0 brings the class back, and it now includes the lexer, the token constants and the reducers as real methods:

1new Compiler()
2    ->load(new File(__DIR__ . '/grammar.pp3'))
3    ->generate()
4        ->withNamespaceName('App\Parser')
5        ->withClassName('LanguageParser')
6        ->save(__DIR__ . '/LanguageParser.php');

See Code Generation.

The `.pp` Format Is No Longer Read

Likelihood Of Impact: Medium

Grammars written in the original Hoa-style .pp format are not supported. A .pp file is still recognized by its extension, so you get a clear error rather than a confusing parse failure.

Rewrite the grammar in one of the formats that are read - see Grammar Syntax.

Grammar Files: What To Check

Likelihood Of Impact: Medium

A grammar file keeps its extension and keeps being read the way it was, so most of them compile unchanged. What no longer works:

The old pragmas are no longer supported. Unification and the error levels are gone; the corresponding behaviour is either the default now or is configured in PHP. Which settings a grammar may carry is listed under Settings.

$file and $state are no longer declared in a reducer. Use $source and $rule, which is an int. See PHP in a Grammar.

Left recursion is now rejected at build time. It never worked at runtime either, but 3.x would let you compile it. Rewrite as a repetition:

1// ✘ rejected
2Expression : Expression() ::T_PLUS:: Number() ;
3
4// ✔
5Expression : Number() (::T_PLUS:: Number())* ;

Reducers returning null mean "no result" and pass the children through. If a rule of yours legitimately produces null, wrap it.

Reducers returning arrays are flattened into the rule above. If you relied on nesting, return an object instead. See Results and Reducers.

Upgrade Guide

Upgrading To 4.0 From 3.x

PHP 8.4 Required

Getters Became Properties

Tokens Are Identified By Number

Channels Replaced "Skipped Tokens"

The Lexer Is Built, Not Configured

The Parser Takes Explicit Arguments

Checking And Trailing Tokens Became One Method

BuilderInterface Became Per-Rule Reducers

Grammar Rule Classes

The Buffer Package Is Gone

The Position and Visitor Packages Are Gone

Sources Are Constructed Directly

Code Generation Is Back

The .pp Format Is No Longer Read

Grammar Files: What To Check

The `.pp` Format Is No Longer Read