Code Generation
Reading a grammar file takes time. Not much - but it is time spent on every single request, doing exactly the same work, to produce exactly the same parser. Generate the parser once instead, commit it, and the compiler never runs in production again.
The Short Version
use Phplrt\Compiler\Compiler;
use Phplrt\Source\File;
new Compiler()
->load(new File(__DIR__ . '/grammar.pp3'))
->generate()
->withNamespaceName('App\Calculator')
->withClassName('SumParser')
->save(__DIR__ . '/SumParser.php');
$parser = new App\Calculator\SumParser();
echo $parser->parse(new Source('2 + 2')); // 4
That is it. The generated file needs phplrt/lexer and phplrt/parser, and
nothing else - phplrt/compiler can go in require-dev.
What Comes Out
Given this grammar:
%skip T_WHITESPACE \s++
%token T_DIGIT \d++
%token T_PLUS \+
Sum -> { return \array_sum($children); }
: Number() (::T_PLUS:: Number())*
;
Number -> { return (int) $children->value; }
: <T_DIGIT>
;
you get this file:
<?php
declare(strict_types=1);
namespace App\Calculator;
/**
* @internal This code has been generated by the phplrt compiler and should not
* be edited by hand
*/
readonly class SumParser extends \Phplrt\Parser\Parser
{
public const int T_WHITESPACE = 0;
public const int T_DIGIT = 1;
public const int T_PLUS = 2;
public function __construct()
{
parent::__construct(
lexer: new \Phplrt\Lexer\Lexer(
pattern: '/\G(?|(?:(?:\s++)(*MARK:0))|(?:(?:\d++)(*MARK:1))|...',
channels: [
'Hidden',
3 => 'Unknown',
],
names: [
'T_WHITESPACE',
'T_DIGIT',
'T_PLUS',
],
),
grammar: [
new \Phplrt\Parser\Grammar\Concatenation([1, 2]),
new \Phplrt\Parser\Grammar\Lexeme(self::T_DIGIT, true),
new \Phplrt\Parser\Grammar\Repetition(3, 0, \INF),
new \Phplrt\Parser\Grammar\Concatenation([4, 1]),
new \Phplrt\Parser\Grammar\Lexeme(self::T_PLUS, false),
],
initial: 0,
reducers: [
0 => self::reduceSum(...),
1 => self::reduceNumber(...),
],
startTokens: [ /* the lookahead table */ ],
matchesEmptyInput: [ /* ... */ ],
presentInTree: [ /* ... */ ],
);
}
private static function reduceSum(\Phplrt\Parser\Context $ctx, mixed $children): mixed
{
return \array_sum($children);
}
private static function reduceNumber(\Phplrt\Parser\Context $ctx, mixed $children): mixed
{
return (int) $children->value;
}
}
A few things worth pointing out.
Tokens become constants. SumParser::T_DIGIT is a real class constant, so
anything reading the token stream can name tokens instead of guessing ids:
if ($token->id === SumParser::T_DIGIT) {
// ...
}
Reducers become methods, named after their rule. Your code is right there - steppable in a debugger, visible in a stack trace, and checked by static analysis like any other file.
The whole lexer is one regular expression. No token list is walked at
runtime; PCRE does the work in a single pass and (*MARK:n) says which
token won.
The lookahead tables are baked in. These are what make the parser quick: before entering a rule, it checks whether the current token could possibly start it, which is one array lookup instead of a doomed descent.
Shaping The Output
generate() returns an immutable object - each method returns a new one, so
order does not matter:
new Compiler()
->load(new File(__DIR__ . '/grammar.pp3'))
->generate()
->withNamespaceName('App\Parser') // namespace App\Parser;
->withClassImport('App\Ast\NumberNode') // use App\Ast\NumberNode;
->withClassImport('App\Ast\Node', as: 'BaseNode') // use App\Ast\Node as BaseNode;
->withClassName('LanguageParser') // class LanguageParser
->save(__DIR__ . '/LanguageParser.php');
save() is the only method that does anything: it writes the code down and
creates the missing directories on the way.
Named Class or Anonymous?
With a class name, you get a declaration:
readonly class LanguageParser extends \Phplrt\Parser\Parser { /* ... */ }
$parser = new App\Parser\LanguageParser();
Without one, the file returns an anonymous parser:
return new readonly class extends \Phplrt\Parser\Parser { /* ... */ };
$parser = require __DIR__ . '/parser.php';
Use the named form for anything autoloaded - it is a normal class, and composer will find it. The anonymous form is handy for a one-off script, or when you do not want the parser in the global class namespace at all.
Getting The Code Without Writing It
The output is Stringable, so you can do what you like with it:
$code = (string) new Compiler()
->load(new File(__DIR__ . '/grammar.pp3'))
->generate()
->withNamespaceName('App\Parser')
->withClassName('LanguageParser');
// run it through php-cs-fixer, diff it against the committed version,
// write it into a phar, whatever you need
A handy trick for CI - fail the build if the committed parser is stale:
$expected = (string) $output;
$actual = \file_get_contents(__DIR__ . '/LanguageParser.php');
if ($expected !== $actual) {
throw new \RuntimeException('Parser is out of date, run "composer build-parser"');
}
Fitting It Into A Project
The usual layout:
resources/grammar/
grammar.pp3
lexemes.pp3
src/Parser/
LanguageParser.php <- generated, committed
bin/
build-parser.php <- runs the generator
{
"scripts": {
"build-parser": "php bin/build-parser.php"
}
}
Run it whenever the grammar changes, and commit the result. Two reasons to commit rather than generate on deploy: the file is what static analysis and your IDE actually see, and a broken grammar fails in a pull request instead of in production.
When Not To Generate
Generation is not always the answer. Read the grammar at runtime when:
- you are still writing it, and regenerating on every change is friction;
- the grammar is user-supplied or assembled at runtime;
- it is a script, run once, where startup cost does not matter.
// Perfectly reasonable for a CLI tool
$parser = new Compiler()
->load(new File($argv[1]))
->getParser();
phplrt Does This To Itself
The .pp3 format is described by a grammar written in .pp3, and the parser
reading your grammar files is generated from it - the same generate() call
you have just seen, committed to the repository.
It has to be that way: reading a grammar file needs a parser, so the parser of
a format cannot be built from the grammar describing it at runtime. The grammar
lives in resources/pp3.pp3 - split with %include into lexemes, statements
and quantifiers - the parser it compiles into lives in
src/Syntax/PP3/PP3Parser.php, and a test compares the two so they cannot
drift apart.
The nice side effect is what you get too: a parser that is built costs a few
milliseconds to construct, and a parser that is generated costs a new.
Writing Your Own Generator
A PHP parser is not the only thing a compiled grammar is good for. An editor
plugin wants the token list; a syntax highlighter wants the pattern; a client
in another language wants the rules. All of that is in CompilerResult, and
the interface for turning it into a file has one method:
public function generate(
CompilerResult $result,
OutputContext $context = new OutputContext(),
): string;
Return a string, and everything else - withNamespaceName(), save(), creating
directories - keeps working as before.
Here is a generator that writes the token list down as JSON, for a highlighter that needs to know the words of a language but not its grammar:
use Phplrt\Compiler\CompilerResult;
use Phplrt\Compiler\Generator\OutputContext;
use Phplrt\Compiler\Generator\OutputGeneratorInterface;
final readonly class TokenListGenerator implements OutputGeneratorInterface
{
public function generate(
CompilerResult $result,
OutputContext $context = new OutputContext(),
): string {
$tokens = [];
foreach ($result->lexer->tokens as $id => $definition) {
$tokens[] = [
'id' => $id,
'name' => $result->lexer->names[$id] ?? null,
'channel' => $result->lexer->channels[$id] ?? 'Default',
];
}
return \json_encode([
'name' => $context->class ?? 'grammar',
'pattern' => $result->lexer->pattern,
'tokens' => $tokens,
], \JSON_PRETTY_PRINT | \JSON_UNESCAPED_SLASHES) . "\n";
}
}
Pass it to generate() in place of the default:
new Compiler()
->load(new File(__DIR__ . '/grammar.pp3'))
->generate(new TokenListGenerator())
->withClassName('Calculator')
->save(__DIR__ . '/calculator.json');
Given this grammar:
%skip T_WHITESPACE \s++
%skip T_COMMENT //[^\n]*+
%token T_NUMBER \d++
%token T_PLUS \+
Sum -> { return \array_sum($children); }
: <T_NUMBER> (::T_PLUS:: <T_NUMBER>)*
;
you get calculator.json:
{
"name": "Calculator",
"pattern": "/\\G(?|(?:(?:\\s++)(*MARK:0))|(?:(?:\\/\\/[^\\n]*+)(*MARK:1))|(?:(?:\\d++)(*MARK:2))|(?:(?:\\+)(*MARK:3))|(?:(?:[^\\s]++)(*MARK:4)))/Ssum",
"tokens": [
{
"id": 0,
"name": "T_WHITESPACE",
"channel": "Hidden"
},
{
"id": 1,
"name": "T_COMMENT",
"channel": "Hidden"
},
{
"id": 2,
"name": "T_NUMBER",
"channel": "Default"
},
{
"id": 3,
"name": "T_PLUS",
"channel": "Default"
},
{
"id": 4,
"name": null,
"channel": "Unknown"
}
]
}
Two things the output shows that the grammar does not say out loud: the
%skip tokens are ordinary tokens on the Hidden channel, and token #4 is
the catch-all the compiler appends so that unrecognized input becomes an
Unknown token instead of an error.
What You Get To Work With
$result->lexer->tokens; // array<int, TokenDefinition>
$result->lexer->names; // array<int, non-empty-string>
$result->lexer->channels; // array<int, non-empty-string>
$result->lexer->pattern; // the compiled regex
$result->lexer->transitions; // which tokens enter a nested lexer
$result->lexer->lexers; // the nested lexers themselves
$result->parser->grammar; // list<RuleInterface> - the rules, by id
$result->parser->initial; // where parsing starts
$result->parser->reducers; // array<int, ReducerInterface>
$result->parser->constants; // ['Sum' => 0, ...] - named rules
$result->parser->startTokens; // the lookahead table
$result->parser->matchesEmptyInput;
$result->parser->presentInTree;
OutputContext carries what the caller asked for - $context->namespace,
$context->class, $context->imports. Use what makes sense for your format
and ignore the rest.
Reporting Problems
Not every grammar can be written into every format. Throw a
GeneratorException when yours cannot express something, so the failure
arrives as a compiler error rather than as broken output:
use Phplrt\Compiler\Exception\GeneratorException;
if ($result->lexer->lexers !== []) {
throw new class ('Nested lexers cannot be written as JSON')
extends GeneratorException {};
}
Adjusting The PHP Output Instead
If you only want to change how the PHP looks, you do not need a generator
of your own. PhpOutputGenerator renders Twig
templates from the compiler's resources/php/ directory, and they are split
per fragment - the file layout, the parser body, the grammar table, the
reducer methods, the lexer and its states - so a change is usually confined
to one small template.