Lexer Builder

This package can be installed separately with composer require phplrt/lexer-builder

A lexer is described before it can read anything: which tokens there are, what each of them looks like, and what the lexer does with them. That description is what the builder is for, and what it compiles into is the lexer itself.

A .pp3 file says the same things in fewer characters, so reach for the builder when the token list is not known in advance - generated from a config file, a database, a plugin system. The last section shows the two spellings side by side.

A First Lexer

Describe the tokens, build the lexer, run it:

 1use Phplrt\Lexer\Builder\LexerBuilder;
 2use Phplrt\Source\StringSource;
 3
 4$builder = new LexerBuilder();
 5$builder->addPattern('\d++', 'T_DIGIT');
 6$builder->addValue('+', 'T_PLUS');
 7$builder->addPattern('\s++', 'T_WHITESPACE');
 8
 9$lexer = $builder->build()
10    ->toLexer();
11
12foreach ($lexer->lex(StringSource::createFromString('23 + 42')) as $token) {
13    echo $token, "\n";
14}

1"23" (T_DIGIT)
2" " (T_WHITESPACE)
3"+" (T_PLUS)
4" " (T_WHITESPACE)
5"42" (T_DIGIT)
6end of input

Two ways to describe a token:

• addPattern('\d++') - a regular expression;
• addValue('+') - a literal string, escaped for you. addValue('+') and addPattern('\+') are the same thing, but the first one is harder to get wrong.

The last token is always end of input. It is how the parser knows the source has been read to the end.

Order Matters

The lexer tries the patterns from top to bottom and takes the first one that matches - not the longest. So this does not do what it looks like:

1$builder->addValue('*', 'T_STAR');
2$builder->addValue('**', 'T_POW');   // never matched!

** is read as two T_STAR tokens, because T_STAR was declared first. Put the longer literal above the shorter one:

1$builder->addValue('**', 'T_POW');   // ✔
2$builder->addValue('*', 'T_STAR');

The same applies to keywords and identifiers - declare if before [a-z]+, or if will always be read as an identifier.

Hiding Whitespace

You almost never want whitespace in a grammar. Mark it as hidden and it is still recognized (so the offsets stay right) but it is left out of the stream entirely:

1$builder->addPattern('\s++')
2    ->hide();
3$builder->addPattern('//[^\n]*+')
4    ->hide(); // line comments too

Note that the token above has no name. A hidden token is not referred to by anything, so naming it is optional.

hide() is shorthand for putting the token on the Hidden channel, and that is the one channel a lexer leaves out by default - see Channels below, and Choosing What Is Reported for how to have the hidden tokens reported after all.

Fragments

A piece of a pattern that keeps coming back is named once and referred to by (?&NAME):

1$builder->addFragment('DIGIT', '[0-9]');
2$builder->addFragment('EXP', '[eE][+-]?(?&DIGIT)++');
3
4$builder->addPattern('(?&DIGIT)++(\.(?&DIGIT)++)?(?&EXP)?', 'T_NUMBER');

The pieces are written into the patterns while the lexer is built, so what it compiles is what they spell out:

(?:[0-9])++(\.(?:[0-9])++)?(?:[eE][+-]?(?:[0-9])++)?

A fragment recognizes nothing on its own and becomes no token. It is written in as a group of its own, so a quantifier after (?&NAME) counts the whole piece; it may be written of another fragment, but not of itself; and it reaches the patterns of every nested lexer, in whatever order it is added.

A name that no fragment is added under is left alone in case the pattern captures a subpattern under it - which is what (?&NAME) means to PCRE - and is reported otherwise:

1$builder->addPattern('\((?<in>[^()]|(?&in))*+\)', 'T_NESTED'); // ✔ left alone
2$builder->addPattern('(?&DIGT)++', 'T_NUMBER');               // ✘ reported

Regex Modifiers

By default, patterns are compiled with S, u, s and m. Add or remove modifiers for the whole lexer:

1use Phplrt\Lexer\Builder\Definition\RegexModifier;
2
3$builder->enable(RegexModifier::Caseless);  // /i
4$builder->disable(RegexModifier::Utf8);     // no /u
5
6$builder->addValue('true', 'T_TRUE');
7// now matches "true", "TRUE" and "True"

Channels

A channel is a label on a token, and it is declared where the token is:

1use Phplrt\Contracts\Lexer\Channel;
2
3$builder->addPattern('\s++')->setChannel(Channel::Hidden);
4$builder->addPattern('\s++')->hide();  // the same, shorter
5$builder->addPattern('\s++')->show();  // back to Default

There are four built-in channels:

Which of them a lexer actually reports is decided when it is built rather than when it is described - see Choosing What Is Reported.

Custom Channels

Hiding a token throws it away. Sometimes you want to keep it, just told apart from the code around it - documentation comments are the usual example. Give it a channel of its own:

1$builder->addPattern('\d++', 'T_DIGIT');
2$builder->addPattern('//[^\n]*+', 'T_COMMENT')->setChannel('comments');
3$builder->addPattern('\s++')->hide();
4
5foreach ($lexer->lex(StringSource::createFromString("1 // hi\n2")) as $token) {
6    echo $token->name, ' on ', $token->channel->name, "\n";
7}

1T_DIGIT on Default
2T_COMMENT on comments
3T_DIGIT on Default

T_COMMENT is in the stream like any other token, and the channel is what tells it apart: a documentation generator reads the comments tokens and ignores the rest.

Reusing The Result

build() gives you a LexerBuilderResult - the compiled description - and toLexer() turns that into a runnable lexer:

1$result = $builder->build();
2
3$result->pattern;  // the single regex the whole lexer compiles down to
4$result->names;    // [0 => 'T_DIGIT', 1 => 'T_PLUS', ...]
5$result->channels; // [2 => 'Hidden', ...]
6
7$lexer = $result->toLexer();

Building is not free - it validates every pattern, drops unreachable tokens and merges everything into one big regular expression. Do it once and keep the lexer around, or better still, generate the code and skip building entirely.

How The Pattern Is Built

All the token definitions compile into a single PCRE pattern, and which token matched is recorded with (*MARK:n):

/\G(?|(?:(?:\d++)(*MARK:0))|(?:(?:\s++)(*MARK:1))|(?:(?:[^\s]++)(*MARK:2)))/Ssum

One pass over the input, one regex, no per-token loop - this is where the lexer's speed comes from. The last branch is the catch-all that produces Unknown tokens.

MarkersRegexGenerator does this, and it is swappable if you ever need a different strategy:

 1use Phplrt\Lexer\Builder\Analysis\RegexConstructionLexerAnalysisPass;
 2use Phplrt\Lexer\Builder\Regex\RegexGeneratorInterface;
 3
 4final class MyRegexGenerator implements RegexGeneratorInterface
 5{
 6    public function generate(array $tokens, array $flags): string
 7    {
 8        // $tokens is a map of token id => TokenDefinition
 9    }
10}
11
12$builder->addAnalysisPass(
13    new RegexConstructionLexerAnalysisPass(new MyRegexGenerator()),
14);

Writing It In A Grammar Instead

Everything above has a shorter spelling in a .pp3 file:

1%fragment DIGIT      [0-9]
2
3%token T_DIGIT       (?&DIGIT)++
4%token T_PLUS        \+
5%skip  T_WHITESPACE  \s++

A token nothing refers to by name does not have to be declared at all - a rule declares it where it reads it, and the two spellings are the two methods above:

1Sum  : <T_DIGIT> "+" <T_DIGIT> ;          // addValue('+')
2Expr : <T_DIGIT> /and|or/ <T_DIGIT> ;     // addPattern('and|or')

The modifiers and the compiler passes are settings of the grammar there, so a grammar carries the way it wants to be compiled:

1%pragma lexer.pcre.flag  Caseless
2%pragma lexer.check      \App\Grammar\MyValidationPass

That is usually where you want to be - see Compiling a Grammar. The builder API is for the cases where the token list is not known in advance: generated from a config file, a database, a plugin system.

What's Next?

• Lexer - reading a source with what this compiles into.
• Nested Lexers - string interpolation, PHP inside HTML and other "a different language starts here" situations.
• Contracts - phplrt/lexer-contracts, for code that needs a lexer without needing this one.