phplrt 4.0

Quick Start

Let's build a parser for a small configuration format from scratch: text in, a tree of objects out.

1// A small config
2name = "phplrt"
3version = 4
4debug = true
1[
2    Property { name: "name",    value: Literal { value: "phplrt" } },
3    Property { name: "version", value: Literal { value: 4 } },
4    Property { name: "debug",   value: Literal { value: true } },
5]

It takes about forty lines of grammar, and along the way you will meet every part of phplrt you are likely to need.

composer require phplrt/runtime
composer require phplrt/compiler --dev

Step 1: Describe The Words

Before anything can be parsed, the text has to be cut into pieces. Create grammar.pp3 and start with the tokens:

1%skip  T_WHITESPACE  \s++
2%skip  T_COMMENT     //[^\n]*+
3
4%token T_BOOLEAN  (?:true|false)\b
5%token T_NAME     [a-zA-Z_][a-zA-Z0-9_]*+
6%token T_STRING   "[^"]*+"
7%token T_NUMBER   \d++
8%token T_EQUAL    =

Each line is a name and a regular expression. %skip is the same as %token, except those tokens never reach the parser - which is exactly what you want for whitespace and comments.

The order matters: the lexer tries the patterns from top to bottom and takes the first one that matches. T_BOOLEAN is above T_NAME for that very reason - the other way round, true would be read as a name.

Step 2: Describe The Sentences

Now the rules. A rule has a name, a :, a body, and an optional ;:

Property : <T_NAME> ::T_EQUAL:: Value() ;

Three things are going on in that line:

  • <T_NAME> reads a token and keeps it;
  • ::T_EQUAL:: reads a token and throws it away - the = has done its job by being there, and nobody needs it afterwards;
  • Value() refers to another rule.

| means "or". The alternatives are tried in order, and the first one that matches wins - so put the more specific ones first:

1Value
2  : String()
3  | Number()
4  | Boolean()
5  ;

A config is a list of properties, so the rule the parsing starts at is a repetition:

1%pragma root Config
2
3Config : Property()* ;

* means "zero or more times", so an empty file is a valid config. There is also + (one or more), ? (optional) and {2,5} (between two and five times). %pragma root names the rule everything else hangs off.

Step 3: Build The Result

At this point the grammar is complete - it can already tell name = 4 from name =. But recognizing text is not the same as getting something out of it. To build a value, attach a reducer: a block of PHP that runs when the rule matches.

Start with the nodes. They are plain objects - no base class, nothing from phplrt in them:

 1namespace App\Ast;
 2
 3final readonly class Property
 4{
 5    public function __construct(
 6        public string $name,
 7        public Literal $value,
 8        public int $offset,
 9    ) {}
10}
11
12final readonly class Literal
13{
14    public function __construct(
15        public mixed $value,
16        public int $offset,
17    ) {}
18}

Now the rules that build them:

 1String -> { return new \App\Ast\Literal(\substr($children->value, 1, -1), $offset); }
 2  : <T_STRING>
 3  ;
 4
 5Number -> { return new \App\Ast\Literal((int) $children->value, $offset); }
 6  : <T_NUMBER>
 7  ;
 8
 9Boolean -> { return new \App\Ast\Literal($children->value === 'true', $offset); }
10  : <T_BOOLEAN>
11  ;

Inside the block, $children holds whatever the rule matched. For String that is the one token it kept, so $children->value is the text "phplrt" - quotes included, which is what substr() takes off.

$offset is where the token starts, in bytes. Keeping it on every node is a habit worth forming: it is what lets a later stage - a validator, a linter, your own error - point at the right place in the file.

For a rule that matched several things, $children is a list:

1Property -> {
2    return new \App\Ast\Property(
3        name: $children[0]->value,
4        value: $children[1],
5        offset: $children[0]->offset,
6    );
7}
8  : <T_NAME> ::T_EQUAL:: Value()
9  ;

Two elements, not three: ::T_EQUAL:: was discarded by the grammar, so the reducer never sees it. And $children[1] is a Literal rather than a token, already built - reducers run bottom-up, innermost rule first.

Value and Config get no reducer at all. A rule without one hands its children up as they are, which is exactly right here: Value is a choice between three rules that already build nodes, and Config is the list of properties itself.

The Whole Grammar

 1%skip  T_WHITESPACE  \s++
 2%skip  T_COMMENT     //[^\n]*+
 3
 4%token T_BOOLEAN  (?:true|false)\b
 5%token T_NAME     [a-zA-Z_][a-zA-Z0-9_]*+
 6%token T_STRING   "[^"]*+"
 7%token T_NUMBER   \d++
 8%token T_EQUAL    =
 9
10// Where the parsing starts
11%pragma root Config
12
13Config : Property()* ;
14
15Property -> {
16    return new \App\Ast\Property(
17        name: $children[0]->value,
18        value: $children[1],
19        offset: $children[0]->offset,
20    );
21}
22  : <T_NAME> ::T_EQUAL:: Value()
23  ;
24
25Value
26  : String()
27  | Number()
28  | Boolean()
29  ;
30
31String -> { return new \App\Ast\Literal(\substr($children->value, 1, -1), $offset); }
32  : <T_STRING>
33  ;
34
35Number -> { return new \App\Ast\Literal((int) $children->value, $offset); }
36  : <T_NUMBER>
37  ;
38
39Boolean -> { return new \App\Ast\Literal($children->value === 'true', $offset); }
40  : <T_BOOLEAN>
41  ;

Step 4: Run It

 1use Phplrt\Compiler\Compiler;
 2use Phplrt\Source\FileSource;
 3use Phplrt\Source\StringSource;
 4
 5$parser = new Compiler()
 6    ->load(FileSource::createFromPathname(__DIR__ . '/grammar.pp3'))
 7    ->getParser();
 8
 9$ast = $parser->parse(StringSource::createFromString(<<<'CONF'
10    // A small config
11    name = "phplrt"
12    version = 4
13    debug = true
14    CONF));
1$ast[0]->name;         // "name"
2$ast[0]->value->value; // "phplrt"
3$ast[0]->offset;       // 18
4
5$ast[1]->value->value; // 4    - an int, because the reducer cast it
6$ast[2]->value->value; // true - a bool, same
7
8$parser->parse(StringSource::createEmpty()); // [] - an empty config is still a config

Notice the two kinds of "source": FileSource reads from disk and StringSource wraps a string you already have. The grammar is a source, and so is the text being parsed - they are the same kind of object.

Step 5: Errors

Feed it something broken, and you get an exception that knows where it happened:

1use Phplrt\Source\StringSource;
2use Phplrt\Source\VirtualSource;
3
4// VirtualSource attaches a name, so errors can point at it
5$parser->parse(VirtualSource::createFromString('config.txt', <<<'CONF'
6    name = "phplrt"
7    version = = 4
8    debug = true
9    CONF));
1error[UnexpectedTokenException]: Syntax error, unexpected "=" (T_EQUAL), T_BOOLEAN, T_STRING or T_NUMBER expected
2 --> config.txt:2:11
31 | name = "phplrt"
42 | version = = 4
5  |           ^
63 | debug = true

If you only want to know whether the input is valid, without building anything, use analyze():

1use Phplrt\Parser\Analysis\Mode;
2
3$parser->analyze(StringSource::createFromString('name = "x"'), Mode::SyntaxCheck);
4// SuccessfulResult - the grammar read the source in full
5
6$parser->analyze(StringSource::createFromString('name ='), Mode::SyntaxCheck);
7// PartialResult - it stopped at the end of what it understood

Note that PartialResult extends SuccessfulResult, so "was it read in full" is !($result instanceof PartialResult).

It also tells you how much of the input is valid and what stands in the way, which is what you want for an editor or a prompt - see Analysing A Source.

Step 6: Compile It Once

Reading grammar.pp3 on every request is wasteful - the grammar does not change between requests. Generate a PHP file instead and commit it:

1use Phplrt\Compiler\Compiler;
2use Phplrt\Source\FileSource;
3
4new Compiler()
5    ->load(FileSource::createFromPathname(__DIR__ . '/grammar.pp3'))
6    ->generate()
7        ->withNamespaceName('App\Config')
8        ->withClassName('CompiledConfigParser')
9        ->save(__DIR__ . '/CompiledConfigParser.php');

You get an ordinary class, with the tokens as constants and every reducer as a real method:

 1namespace App\Config;
 2
 3class CompiledConfigParser implements \Phplrt\Contracts\Parser\ParserInterface
 4{
 5    public const int T_WHITESPACE = 0;
 6    public const int T_COMMENT = 1;
 7    public const int T_BOOLEAN = 2;
 8    // ...
 9
10    public function __construct()
11    {
12        $this->lexer = /* the whole lexer, inlined */;
13        $this->parser = /* the whole grammar, inlined */;
14    }
15
16    private static function reduceString(\Phplrt\Parser\Context $ctx, mixed $children): mixed
17    {
18        // The variables below are declared by the compiler
19        $offset = $ctx->begin;
20
21        return new \App\Ast\Literal(\substr($children->value, 1, -1), $offset);
22    }
23
24    // ...
25}

Which you use like any other class - no compiler, no grammar file:

1$parser = new App\Config\CompiledConfigParser();
2
3$parser->parse(StringSource::createFromString('debug = true'))[0]->name; // "debug"

Add the generation call to your build script or a console command, and you are done.

Step 7: Extend The Parser

So far the config can only say what is written in it. Real formats reach outside: environment variables, includes, a base directory, a registry of keys that are allowed at all.

The generated parser is an ordinary class, so extend it - and the grammar can reach whatever you add, through $this.

Add references to the grammar:

 1%token T_REFERENCE  \$\{([a-zA-Z_][a-zA-Z0-9_]*+)\}
 2
 3Value
 4  : String()
 5  | Number()
 6  | Boolean()
 7  | Reference()
 8  ;
 9
10// $this is the parser itself, so the value comes from the subclass
11Reference -> { return $this->reference($children->captures[0], $offset, $source); }
12  : <T_REFERENCE>
13  ;

The pattern has a capturing group, and $children->captures[0] is what that group matched - the name alone, without the ${} around it. More on captures in What A Token Is.

Regenerate, and notice what the compiler did with the new rule:

1private function reduceReference(\Phplrt\Parser\Context $ctx, mixed $children): mixed
2{
3    // The variables below are declared by the compiler
4    $source = $ctx->source;
5    $offset = $ctx->begin;
6
7    return $this->reference($children->captures[0], $offset, $source);
8}

A reducer that mentions $this becomes a non-static method, and the grammar table refers to it as $this->reduceReference(...) instead of self::reduceString(...). The compiler works this out per reducer, so you do not configure anything.

Now subclass and fill in reference():

 1namespace App\Config;
 2
 3use App\Ast\Literal;
 4use Phplrt\Contracts\Source\ReadableInterface;
 5use Phplrt\Exception\ErrorPrinter;
 6
 7final class UnknownVariableException extends \InvalidArgumentException
 8{
 9    public function __construct(
10        string $name,
11        private readonly ReadableInterface $source,
12        private readonly int $offset,
13        private readonly int $length,
14    ) {
15        parent::__construct(\sprintf('Unknown variable "%s"', $name));
16    }
17
18    public function __toString(): string
19    {
20        return (string) new ErrorPrinter()
21            ->print($this)
22            ->withSource($this->source)
23            ->withInterval($this->offset, $this->length);
24    }
25}
26
27final readonly class ConfigParser extends CompiledConfigParser
28{
29    /**
30     * @param array<non-empty-string, string> $variables
31     */
32    public function __construct(
33        private array $variables = [],
34    ) {
35        parent::__construct();
36    }
37
38    protected function reference(string $name, int $offset, ReadableInterface $source): Literal
39    {
40        if (!isset($this->variables[$name])) {
41            throw new UnknownVariableException($name, $source, $offset, \strlen($name) + 3);
42        }
43
44        return new Literal($this->variables[$name], $offset);
45    }
46}
1$parser = new ConfigParser(['APP_ENV' => 'prod']);
2
3$parser->parse(StringSource::createFromString('env = ${APP_ENV}'))[0]
4    ->value->value; // "prod"

The grammar stays a description of the language: ${...} is a reference in this format whatever your variables happen to be. What a reference resolves to is a decision, and decisions live in PHP. Each instance carries its own data, so two parsers with different variables are two objects.

And because the reducer handed reference() the offset and the source, an error from your code reads exactly like one from the parser:

1error[UnknownVariableException]: Unknown variable "NOPE"
2 --> config.txt:2:7
31 | name = "phplrt"
42 | env = ${NOPE}
5  |       ^^^^^^^
63 |

print() takes the exception and reads the message and the name off it, so only the place has to be pointed at - see Error Reporting. Subclassing has a few rules of its own, and they are collected in Best Practices.

What's Next?