Lexer

This package can be installed separately with composer require phplrt/lexer

The lexer is the first half of reading source code: it turns a stream of characters into a stream of tokens. 23 + 42 becomes "a number, a plus, a number" - and the parser never has to look at a single character again.

What it reads is fixed by the time it exists: a lexer is one compiled regular expression plus a couple of tables, produced by the lexer builder or by the grammar compiler. This page is about running one.

Reading A Source

lex() returns a list of tokens. Every token knows what it is, what it says and where it was:

1foreach ($lexer->lex(StringSource::createFromString('23 + 42')) as $token) {
2    $token->id;      // int    - 0, the position of its definition
3    $token->name;    // string - "T_DIGIT"
4    $token->value;   // string - "23"
5    $token->offset;  // int    - 0, in bytes
6    $token->size;    // int    - 2, in bytes
7    $token->channel; // Channel::Default
8}

The last token is always end of input. It is how the parser knows the source has been read to the end.

You can also start somewhere other than the beginning:

1$lexer->lex(StringSource::createFromString('12 34'), offset: 3);
2// only "34" and the end of input

What is on a token, and what to do with it, is What A Token Is.

Channels

A lexer reports every channel except the ones it is told to skip, and by default that is Hidden alone:

1use Phplrt\Lexer\Lexer;
2
3Lexer::DEFAULT_SKIP_CHANNELS; // [Channel::Hidden]

Nothing stands between the lexer and the grammar, so whatever the lexer reports reaches the parser. What the parser reads is another matter: it steps over a token on a channel of your own unless a rule names it. Skipping it here means no rule can ask for it at all.

The list is a setting of the lexer, so it goes where the lexer is put together:

 1use Phplrt\Lexer\Builder\Transformer\RuntimeLexerTransformer;
 2
 3$result = $builder->build();
 4
 5// Reports everything but the hidden tokens
 6$lexer = $result->toLexer();
 7
 8// Reports everything, hidden tokens included
 9$verbose = new RuntimeLexerTransformer(skip: [])
10    ->transform($result);

Channels are matched by name, so a custom one is skipped by naming it - the instance you construct does not have to be the one the token carries:

1use Phplrt\Contracts\Lexer\Channel;
2use Phplrt\Contracts\Lexer\UserDefinedChannel;
3
4$parsing = new RuntimeLexerTransformer(skip: [
5    Channel::Hidden,
6    new UserDefinedChannel('comments'),
7])->transform($result);

One description therefore gives you as many lexers as there are readers of the source: one leaving the comments out for the parser, and one reporting them for the tool that wants them.

That is where the list ends up anyway - Lexer takes it as an argument of its own, so a lexer assembled without the builder is configured the same way:

1use Phplrt\Lexer\Lexer;
2
3$lexer = new Lexer(
4    pattern: $result->pattern,
5    channels: $result->channels,
6    names: $result->names,
7    skip: [],
8);

Skipping happens while the source is being read, so a token nobody is going to see is not built in the first place.

Unrecognized Input

The lexer never fails on text it does not recognize. It emits a token on the Unknown channel instead and carries on:

 1$builder = new LexerBuilder();
 2$builder->addPattern('\d++', 'T_DIGIT');
 3$builder->addPattern('\s++')
 4    ->hide();
 5    
 6$lexer = $builder->build()
 7    ->toLexer();
 8
 9foreach ($lexer->lex(StringSource::createFromString('12 @ 34')) as $token) {
10    echo $token . "\n";
11}
12
13// "12" (T_DIGIT)
14// "@" (unknown token)
15// "34" (T_DIGIT)
16// end of input

This is deliberate: a lexer that stops at the first strange character can only report one problem, while an editor or a linter usually wants to see the whole file. The parser is the one that decides an unknown token is an error, and it does so with a message pointing at the exact spot.

System Tokens

Two tokens are produced by the lexer rather than declared, and they use negative identifiers so they can never collide with yours:

What's Next?

• Lexer Builder - describing the tokens this reads, and the channels they go on.
• Nested Lexers - string interpolation, PHP inside HTML and other "a different language starts here" situations.
• Contracts - phplrt/lexer-contracts, for code that needs a lexer without needing this one.