phplrt 4.0

Quick Start

Let's build a parser for a small configuration format from scratch: text in, a tree of objects out.

1// A small config
2name = "phplrt"
3version = 4
4debug = true
1[
2    Property { name: "name",    value: Literal { value: "phplrt" } },
3    Property { name: "version", value: Literal { value: 4 } },
4    Property { name: "debug",   value: Literal { value: true } },
5]

It takes about forty lines of grammar, and along the way you will meet every part of phplrt you are likely to need.

composer require phplrt/runtime
composer require phplrt/compiler --dev

Step 1: Describe The Words

Before anything can be parsed, the text has to be cut into pieces. Create grammar.pp3 and start with the tokens:

1%skip  T_WHITESPACE  \s++
2%skip  T_COMMENT     //[^\n]*+
3
4%token T_BOOLEAN  (?:true|false)\b
5%token T_NAME     [a-zA-Z_][a-zA-Z0-9_]*+
6%token T_STRING   "[^"]*+"
7%token T_NUMBER   \d++
8%token T_EQUAL    =

Each line is a name and a regular expression. %skip is the same as %token, except those tokens never reach the parser - which is exactly what you want for whitespace and comments.

The order matters: the lexer tries the patterns from top to bottom and takes the first one that matches. T_BOOLEAN is above T_NAME for that very reason - the other way round, true would be read as a name.

Step 2: Describe The Sentences

Now the rules. A rule has a name, a :, a body, and an optional ;:

Property : <T_NAME> ::T_EQUAL:: Value() ;

Three things are going on in that line:

  • <T_NAME> reads a token and keeps it;
  • ::T_EQUAL:: reads a token and throws it away - the = has done its job by being there, and nobody needs it afterwards;
  • Value() refers to another rule.

| means "or". The alternatives are tried in order, and the first one that matches wins - so put the more specific ones first:

1Value
2  : String()
3  | Number()
4  | Boolean()
5  ;

A config is a list of properties, so the rule the parsing starts at is a repetition:

1%pragma root Config
2
3Config : Property()* ;

* means "zero or more times", so an empty file is a valid config. There is also + (one or more), ? (optional) and {2,5} (between two and five times). %pragma root names the rule everything else hangs off.

Step 3: Say What To Build

At this point the grammar is complete - it can already tell name = 4 from name =. But recognizing text is not the same as getting something out of it. To build a value, attach a reducer: a block of PHP that runs when the rule matches.

Start with the nodes. They are plain objects - no base class, nothing from phplrt in them:

 1namespace App\Ast;
 2
 3final readonly class Property
 4{
 5    public function __construct(
 6        public string $name,
 7        public Literal $value,
 8        public int $offset,
 9    ) {}
10}
11
12final readonly class Literal
13{
14    public function __construct(
15        public mixed $value,
16        public int $offset,
17    ) {}
18}

Now the rules that build them:

 1String -> { return new \App\Ast\Literal(\substr($children->value, 1, -1), $offset); }
 2  : <T_STRING>
 3  ;
 4
 5Number -> { return new \App\Ast\Literal((int) $children->value, $offset); }
 6  : <T_NUMBER>
 7  ;
 8
 9Boolean -> { return new \App\Ast\Literal($children->value === 'true', $offset); }
10  : <T_BOOLEAN>
11  ;

Inside the block, $children holds whatever the rule matched. For String that is the one token it kept, so $children->value is the text "phplrt" - quotes included, which is what substr() takes off.

$offset is where the token starts, in bytes. Keeping it on every node is a habit worth forming: it is what lets a later stage - a validator, a linter, your own error - point at the right place in the file.

For a rule that matched several things, $children is a list:

1Property -> {
2    return new \App\Ast\Property(
3        name: $children[0]->value,
4        value: $children[1],
5        offset: $children[0]->offset,
6    );
7}
8  : <T_NAME> ::T_EQUAL:: Value()
9  ;

Two elements, not three: ::T_EQUAL:: was discarded by the grammar, so the reducer never sees it. And $children[1] is a Literal rather than a token, already built - reducers run bottom-up, innermost rule first.

Value and Config get no reducer at all. A rule without one hands its children up as they are, which is exactly right here: Value is a choice between three rules that already build nodes, and Config is the list of properties itself.

The Whole Grammar

 1%skip  T_WHITESPACE  \s++
 2%skip  T_COMMENT     //[^\n]*+
 3
 4%token T_BOOLEAN  (?:true|false)\b
 5%token T_NAME     [a-zA-Z_][a-zA-Z0-9_]*+
 6%token T_STRING   "[^"]*+"
 7%token T_NUMBER   \d++
 8%token T_EQUAL    =
 9
10// Where the parsing starts
11%pragma root Config
12
13Config : Property()* ;
14
15Property -> {
16    return new \App\Ast\Property(
17        name: $children[0]->value,
18        value: $children[1],
19        offset: $children[0]->offset,
20    );
21}
22  : <T_NAME> ::T_EQUAL:: Value()
23  ;
24
25Value
26  : String()
27  | Number()
28  | Boolean()
29  ;
30
31String -> { return new \App\Ast\Literal(\substr($children->value, 1, -1), $offset); }
32  : <T_STRING>
33  ;
34
35Number -> { return new \App\Ast\Literal((int) $children->value, $offset); }
36  : <T_NUMBER>
37  ;
38
39Boolean -> { return new \App\Ast\Literal($children->value === 'true', $offset); }
40  : <T_BOOLEAN>
41  ;

Step 4: Run It

 1use Phplrt\Compiler\Compiler;
 2use Phplrt\Source\File;
 3use Phplrt\Source\Source;
 4
 5$parser = new Compiler()
 6    ->load(new File(__DIR__ . '/grammar.pp3'))
 7    ->getParser();
 8
 9$ast = $parser->parse(new Source(<<<'CONF'
10    // A small config
11    name = "phplrt"
12    version = 4
13    debug = true
14    CONF));
1$ast[0]->name;         // "name"
2$ast[0]->value->value; // "phplrt"
3$ast[0]->offset;       // 18
4
5$ast[1]->value->value; // 4    - an int, because the reducer cast it
6$ast[2]->value->value; // true - a bool, same
7
8$parser->parse(new Source('')); // [] - an empty config is still a config

Notice the two kinds of "source": File reads from disk, Source wraps a string you already have. The grammar is a source, and so is the text being parsed - they are the same kind of object.

Step 5: Errors

Feed it something broken, and you get an exception that knows where it happened:

1use Phplrt\Source\VirtualFile;
2
3// VirtualFile is a string that also has a name, so errors can point at it
4$parser->parse(new VirtualFile('config.txt', <<<'CONF'
5    name = "phplrt"
6    version = = 4
7    debug = true
8    CONF));
1error[UnexpectedTokenException]: Syntax error, unexpected "=" (T_EQUAL), one of T_BOOLEAN, T_STRING, T_NUMBER expected
2 --> config.txt:2:11
3  |
41 | name = "phplrt"
52 | version = = 4
6  |           ^
73 | debug = true

If you only want to know whether the input is valid, without building anything, use analyze():

1use Phplrt\Parser\Analysis\Mode;
2use Phplrt\Parser\Analysis\Result\SuccessfulResult;
3
4$parser->analyze(new Source('name = "x"'), Mode::SyntaxCheck) instanceof SuccessfulResult; // true
5$parser->analyze(new Source('name ='), Mode::SyntaxCheck) instanceof SuccessfulResult;     // false

It also tells you how much of the input is valid and what stands in the way, which is what you want for an editor or a prompt - see Analysing A Source.

Step 6: Compile It Once

Reading grammar.pp3 on every request is wasteful - the grammar does not change between requests. Generate a PHP file instead and commit it:

1use Phplrt\Compiler\Compiler;
2use Phplrt\Source\File;
3
4new Compiler()
5    ->load(new File(__DIR__ . '/grammar.pp3'))
6    ->generate()
7        ->withNamespaceName('App\Config')
8        ->withClassName('CompiledConfigParser')
9        ->save(__DIR__ . '/CompiledConfigParser.php');

You get an ordinary class, with the tokens as constants and every reducer as a real method:

 1namespace App\Config;
 2
 3readonly class CompiledConfigParser extends \Phplrt\Parser\Parser
 4{
 5    public const int T_WHITESPACE = 0;
 6    public const int T_COMMENT = 1;
 7    public const int T_BOOLEAN = 2;
 8    // ...
 9
10    public function __construct()
11    {
12        parent::__construct(/* the whole grammar, inlined */);
13    }
14
15    private static function reduceString(\Phplrt\Parser\Context $ctx, mixed $children): mixed
16    {
17        // The variables below are declared by the compiler
18        $offset = $ctx->token->offset;
19
20        return new \App\Ast\Literal(\substr($children->value, 1, -1), $offset);
21    }
22
23    // ...
24}

Which you use like any other class - no compiler, no grammar file:

1$parser = new App\Config\CompiledConfigParser();
2
3$parser->parse(new Source('debug = true'))[0]->name; // "debug"

Add the generation call to your build script or a console command, and you are done.

Step 7: Give The Parser Something To Work With

So far the config can only say what is written in it. Real formats reach outside: environment variables, includes, a base directory, a registry of keys that are allowed at all.

The generated parser is an ordinary class, so extend it - and the grammar can reach whatever you add, through $this.

Add references to the grammar:

 1%token T_REFERENCE  \$\{([a-zA-Z_][a-zA-Z0-9_]*+)\}
 2
 3Value
 4  : String()
 5  | Number()
 6  | Boolean()
 7  | Reference()
 8  ;
 9
10// $this is the parser itself, so the value comes from the subclass
11Reference -> { return $this->reference($children->captures[0], $offset, $source); }
12  : <T_REFERENCE>
13  ;

The pattern has a capturing group, and $children->captures[0] is what that group matched - the name alone, without the ${} around it. More on captures in Tokens and Channels.

Regenerate, and notice what the compiler did with the new rule:

1private function reduceReference(\Phplrt\Parser\Context $ctx, mixed $children): mixed
2{
3    // The variables below are declared by the compiler
4    $source = $ctx->source;
5    $offset = $ctx->token->offset;
6
7    return $this->reference($children->captures[0], $offset, $source);
8}

A reducer that mentions $this becomes a non-static method, and the grammar table refers to it as $this->reduceReference(...) instead of self::reduceString(...). The compiler works this out per reducer, so you do not configure anything.

Now subclass and fill in reference():

 1namespace App\Config;
 2
 3use App\Ast\Literal;
 4use Phplrt\Contracts\Source\ReadableInterface;
 5use Phplrt\Exception\ErrorPrinter;
 6
 7final class UnknownVariableException extends \InvalidArgumentException {}
 8
 9final readonly class ConfigParser extends CompiledConfigParser
10{
11    /**
12     * @param array<non-empty-string, string> $variables
13     */
14    public function __construct(
15        private array $variables = [],
16    ) {
17        parent::__construct();
18    }
19
20    protected function reference(string $name, int $offset, ReadableInterface $source): Literal
21    {
22        if (!isset($this->variables[$name])) {
23            throw new UnknownVariableException((string) new ErrorPrinter()
24                ->print($source, $offset, \strlen($name) + 3)
25                ->withMessage(\sprintf('Unknown variable "%s"', $name))
26                ->withClass('UnknownVariableException'));
27        }
28
29        return new Literal($this->variables[$name], $offset);
30    }
31}
1$parser = new ConfigParser(['APP_ENV' => 'prod']);
2
3$parser->parse(new Source('env = ${APP_ENV}'))[0]
4    ->value->value; // "prod"

The grammar stays a description of the language: ${...} is a reference in this format whatever your variables happen to be. What a reference resolves to is a decision, and decisions live in PHP. Each instance carries its own data, so two parsers with different variables are two objects.

And because the reducer handed reference() the offset and the source, an error from your code reads exactly like one from the parser:

1error[UnknownVariableException]: Unknown variable "NOPE"
2 --> config.txt:2:7
3  |
41 | name = "phplrt"
52 | env = ${NOPE}
6  |       ^^^^^^^
73 |

ErrorPrinter renders any offset in any source - see Error Reporting. Subclassing has a few rules of its own, and they are collected in Best Practices.

Where To Go Next