Tokens and Channels

A token is the smallest thing a lexer produces. It is immutable, and it carries five pieces of information.

1foreach ($lexer->lex(new Source('23 + 42')) as $token) {
2    $token->id;      // 0       - which definition matched
3    $token->name;    // T_DIGIT - its human-readable name, or null
4    $token->value;   // "23"    - the exact text that matched
5    $token->offset;  // 0       - where it starts, in bytes
6    $token->size;    // 2       - how long it is, in bytes
7    $token->channel; // Channel::Default
8}

Identifiers, Not Names

Tokens are compared by identifier, never by name:

1if ($token->id === MyParser::T_DIGIT) {
2    // ...
3}

The identifier is just the position of the token's definition in the lexer, assigned when the lexer is built. Names exist for you and for error messages; the parser does not use them at all, which is why a token is allowed to have no name:

$builder->addPattern('\s++')->hide(); // anonymous, and that is fine

System tokens use negative identifiers, so they can never collide with yours:

Offsets

offset and size are counted in bytes, not in characters, so the two of them address the fragment directly:

1$content = $source->content;
2
3// The exact fragment the token was read from
4$text = \substr($content, $token->offset, $token->size);
5
6// Where the next token starts
7$next = $token->offset + $token->size;

For an ordinary token size is simply strlen($value). It differs only for a token that entered a nested lexer, which is as large as everything that lexer read.

Printing

Tokens are Stringable, and they print in a form meant for error messages:

echo $token;

1"23" (T_DIGIT)      // a named token
2"@" (unknown token) // unrecognized input
3end of input        // the terminal token

Long values are cut off, and control characters are escaped, so you can put a token straight into a message without worrying about what is in it.

Channels

A channel is a label on a token. The lexer decides which channels it reports, and a token on any other channel is read like any other - so the offsets stay right - and then stepped over.

There are four built-in channels:

1use Phplrt\Contracts\Lexer\Channel;
2
3$builder->addPattern('\s++')->setChannel(Channel::Hidden);
4$builder->addPattern('\s++')->hide();  // the same, shorter
5$builder->addPattern('\s++')->show();  // back to Default

Custom Channels

Hiding a token throws it away. Sometimes you want to keep it, just told apart from the code around it - documentation comments are the usual example. Give it a channel of its own:

1$builder->addPattern('\d++', 'T_DIGIT');
2$builder->addPattern('//[^\n]*+', 'T_COMMENT')->setChannel('comments');
3$builder->addPattern('\s++')->hide();
4
5foreach ($lexer->lex(new Source("1 // hi\n2")) as $token) {
6    echo $token->name, ' on ', $token->channel->name, "\n";
7}

1T_DIGIT on Default
2T_COMMENT on comments
3T_DIGIT on Default

T_COMMENT is in the stream like any other token, and the channel is what tells it apart: a documentation generator reads the comments tokens and ignores the rest.

Choosing What Is Reported

A lexer reports every channel except the ones it is told to skip, and by default that is Hidden alone:

1use Phplrt\Lexer\Lexer;
2
3Lexer::DEFAULT_SKIP_CHANNELS; // [Channel::Hidden]

Nothing stands between the lexer and the grammar, so this is also the answer to "what does the parser see": whatever the lexer reports.

The list is a setting of the lexer, so it goes where the lexer is put together:

 1use Phplrt\Lexer\Builder\Transformer\RuntimeLexerTransformer;
 2
 3$result = $builder->build();
 4
 5// Reports everything but the hidden tokens
 6$lexer = $result->toLexer();
 7
 8// Reports everything, hidden tokens included
 9$verbose = new RuntimeLexerTransformer(skip: [])
10    ->transform($result);

Channels are matched by name, so a custom one is skipped by naming it - the instance you construct does not have to be the one the token carries:

1use Phplrt\Contracts\Lexer\Channel;
2use Phplrt\Contracts\Lexer\UserDefinedChannel;
3
4$parsing = new RuntimeLexerTransformer(skip: [
5    Channel::Hidden,
6    new UserDefinedChannel('comments'),
7])->transform($result);

One description therefore gives you as many lexers as there are readers of the source: one leaving the comments out for the parser, and one reporting them for the tool that wants them.

That is where the list ends up anyway - Lexer takes it as an argument of its own, so a lexer assembled without the builder is configured the same way:

1use Phplrt\Lexer\Lexer;
2
3$lexer = new Lexer(
4    pattern: $result->pattern,
5    channels: $result->channels,
6    names: $result->names,
7    skip: [],
8);

Skipping happens while the source is being read, so a token nobody is going to see is not built in the first place.

Captures

If a token's pattern has capturing groups, whatever they matched is on the token:

1$builder->addPattern('"([^"]*)"', 'T_STRING');
2$builder->addPattern('(\d++)\.(\d++)', 'T_FLOAT');
3
4// "hi" 3.14
5$string->captures; // ["hi"]
6$float->captures;  // ["3", "14"]

Captures are numbered per token, starting at zero - the first group of this token is captures[0], no matter how many groups the tokens above it have.

This saves you from parsing the value twice:

1// Instead of trimming the quotes off $token->value by hand
2$content = $token->captures[0];

A group that matched nothing still counts, so the numbering stays stable:

1$builder->addPattern('(\+|-)?(\d++)', 'T_NUMBER');
2
3// "42"  => captures: ["", "42"]
4// "-42" => captures: ["-", "42"]

The Interface

Write against the contract, not the implementation:

1use Phplrt\Contracts\Lexer\TokenInterface;
2
3function describe(TokenInterface $token): string
4{
5    return \sprintf('%s at %d', $token->name ?? 'anonymous', $token->offset);
6}

Phplrt\Lexer\Token\Token is the standard implementation; TokenEmbedding extends it for nested lexers.