Lexer Contracts

This document describes a common interface for lexical analysis - turning a source into the tokens it consists of - together with the interfaces for the tokens themselves, for the channels those tokens are marked with, and for the errors raised along the way.

The goal is to let anything that consumes tokens - a parser, a syntax highlighter, a linter, a formatter - depend on the ability to read them rather than on a particular lexer.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

1. Specification

1.1 Lexer

A lexer is an object implementing LexerInterface.

The lex() method takes a source implementing ReadableInterface and returns an iterable of objects implementing TokenInterface.
The returned value MAY be of any iterable kind, an array and a generator among them.
The $offset argument is counted in bytes from the beginning of the source, and the analysis MUST begin there rather than at the beginning of the source. It defaults to 0.
An implementation MUST NOT change its own state during the analysis, so that one lexer can be used concurrently.

1use Phplrt\Contracts\Lexer\LexerInterface;
2use Phplrt\Contracts\Source\ReadableInterface;
3
4function highlight(LexerInterface $lexer, ReadableInterface $source): string
5{
6    foreach ($lexer->lex($source) as $token) {
7        // ...
8    }
9}

1.2 Tokens

A token is an object implementing TokenInterface. It is the smallest meaningful unit a source consists of.

An implementation MUST be immutable.
$offset is counted in bytes from the beginning of the source, starting at TokenInterface::MIN_OFFSET, and MUST point at the start of the token.
$size MUST be the size of the source fragment the token has been read from, so that the position right after the token is $offset + $size.
$value MUST be the exact source fragment matched, without normalization or transformation of any kind. Any such conversion SHOULD be performed during the syntax analysis.
$id identifies the token type. A significant token SHOULD be identified by a number greater than or equal to zero, and a system one - the end of input, an error - by a negative number.
$name is the human-readable name of the token type, and is null for a token type that has none.
A token is Stringable.

$size is not the length of $value. An ordinary token is as large as its own value, while a token that read its fragment through a lexer of its own is as large as that lexer consumed, however short the value it produced.

1.3 Channels

A channel is a tag a token or a group of tokens is marked with, so that these tokens can be told apart from the rest of a stream. A channel is an object implementing ChannelInterface, whose $name MUST be a non-empty string.

The Channel enum is the basic set of channels:

Case	Meaning
`Default`	A significant token
`Hidden`	A token that MUST be ignored
`Unknown`	A fragment that has not been recognized
`EndOfInput`	The terminal token

A stream MUST contain at most one token of the EndOfInput channel.
An implementation MAY mark a token with a channel of its own, which MUST NOT be a member of the basic set. UserDefinedChannel is such a channel.

1.4 Errors

Every exception thrown by a lexer MUST implement LexerExceptionInterface.
An error that occurs before the analysis begins - a source that cannot be recognized, settings that contain errors - is a plain LexerExceptionInterface.
An error that occurs after the analysis has begun, and therefore indicates a problem in the analyzed source, MUST implement RuntimeExceptionInterface, which carries the source it occurred in and the token it occurred on.

2. Package

The interfaces described here are provided as part of the phplrt/lexer-contracts package:

composer require phplrt/lexer-contracts

It contains no lexer, and requires nothing but PHP and phplrt/source-contracts.

A package implementing these interfaces provides the virtual package phplrt/lexer-contracts-implementation, so an application can ask for an implementation without naming one:

composer require phplrt/lexer-contracts-implementation

phplrt/lexer provides it: Phplrt\Lexer\Lexer implements LexerInterface and Phplrt\Lexer\Token\Token implements TokenInterface.

The package ships a test declaring an anonymous implementation of every interface in it. Changing that test is allowed only in a major release, so no member can be added to any of these interfaces in a minor version.

3. Interfaces

3.1 `Phplrt\Contracts\Lexer\LexerInterface`

 1<?php
 2
 3declare(strict_types=1);
 4
 5namespace Phplrt\Contracts\Lexer;
 6
 7use Phplrt\Contracts\Lexer\Exception\LexerExceptionInterface;
 8use Phplrt\Contracts\Lexer\Exception\RuntimeExceptionInterface;
 9use Phplrt\Contracts\Source\ReadableInterface;
10
11/**
12 * Converts a source into the tokens it consists of.
13 */
14interface LexerInterface
15{
16    /**
17     * Performs lexical analysis of the given source and returns its tokens.
18     *
19     * An implementation MUST NOT change its own state during the analysis, so
20     * that the same lexer can be used in asynchronous and parallel computing.
21     *
22     * @param int<0, max> $offset the offset in bytes from the beginning of the
23     *        source the analysis starts at
24     * @return iterable<array-key, TokenInterface> the analyzed tokens
25     * @throws LexerExceptionInterface if the given source cannot be recognized
26     *         or the lexer settings contain errors
27     * @throws RuntimeExceptionInterface if the analyzed source contains errors
28     */
29    public function lex(ReadableInterface $source, int $offset = 0): iterable;
30}

3.2 `Phplrt\Contracts\Lexer\TokenInterface`

 1<?php
 2
 3declare(strict_types=1);
 4
 5namespace Phplrt\Contracts\Lexer;
 6
 7/**
 8 * A single lexical token, which is the smallest meaningful unit a source
 9 * consists of.
10 *
11 * The offset of a token MUST be counted in bytes from the beginning of the
12 * source, starting at zero, and its size MUST be that of the source fragment
13 * the token has been read from, so that the position right after the token is
14 * the sum of the two.
15 *
16 * An implementation MUST be immutable.
17 *
18 * @readonly
19 */
20interface TokenInterface extends \Stringable
21{
22    /**
23     * The minimal offset a token is allowed to have.
24     *
25     * @var int<0, max>
26     */
27    public const int MIN_OFFSET = 0;
28
29    /**
30     * The identifier of the token type, by which the kinds of the tokens are
31     * told apart from one another.
32     *
33     * A significant token SHOULD be identified by a number greater than or
34     * equal to zero, while a system one, like the end of input or an error,
35     * SHOULD be identified by a negative number.
36     */
37    public int $id {
38        get;
39    }
40
41    /**
42     * The human-readable name of the token type, or {@see null} in case the
43     * token type has no name of its own.
44     *
45     * @var non-empty-string|null
46     */
47    public ?string $name {
48        get;
49    }
50
51    /**
52     * The channel the token belongs to.
53     */
54    public ChannelInterface $channel {
55        get;
56    }
57
58    /**
59     * The offset in bytes from the beginning of the source the token
60     * starts at.
61     *
62     * @var int<0, max>
63     */
64    public int $offset {
65        get;
66    }
67
68    /**
69     * The size of the source fragment the token has been read from, in bytes.
70     *
71     * An ordinary token is as large as its own value, while a token reading a
72     * fragment using a lexer of its own is as large as that lexer has read, no
73     * matter how short its own value is.
74     *
75     * @var int<0, max>
76     */
77    public int $size {
78        get;
79    }
80
81    /**
82     * The exact source fragment matched by the lexer.
83     *
84     * The value MUST be the original fragment, without any normalization or
85     * transformation, which SHOULD be performed during the syntax analysis.
86     */
87    public string $value {
88        get;
89    }
90}

3.3 `Phplrt\Contracts\Lexer\ChannelInterface`

 1<?php
 2
 3declare(strict_types=1);
 4
 5namespace Phplrt\Contracts\Lexer;
 6
 7/**
 8 * A tag a token or a group of tokens is marked with, so that these tokens can
 9 * be told apart from the rest of a stream.
10 */
11interface ChannelInterface
12{
13    /**
14     * The name of the channel.
15     *
16     * @var non-empty-string
17     */
18    public string $name {
19        get;
20    }
21}

3.4 `Phplrt\Contracts\Lexer\Channel`

 1<?php
 2
 3declare(strict_types=1);
 4
 5namespace Phplrt\Contracts\Lexer;
 6
 7/**
 8 * The basic set of the token channels.
 9 *
10 * An implementation MAY mark a token with a channel of its own, which MUST
11 * NOT be a member of this set.
12 */
13enum Channel implements ChannelInterface
14{
15    /**
16     * The channel of the significant tokens.
17     */
18    case Default;
19
20    /**
21     * The channel of the tokens that MUST be ignored.
22     */
23    case Hidden;
24
25    /**
26     * The channel of the tokens that have not been recognized.
27     */
28    case Unknown;
29
30    /**
31     * The channel of the terminal token.
32     *
33     * A stream MUST contain at most one token of this channel.
34     */
35    case EndOfInput;
36
37    /**
38     * The channel of the significant tokens.
39     */
40    public const self DEFAULT = self::Default;
41
42    /**
43     * Returns every channel of this set, indexed by its own name.
44     *
45     * @return non-empty-array<non-empty-string, Channel>
46     */
47    public static function names(): array
48    {
49        $result = [];
50
51        foreach (self::cases() as $case) {
52            $result[$case->name] = $case;
53        }
54
55        return $result;
56    }
57}

3.5 `Phplrt\Contracts\Lexer\UserDefinedChannel`

 1<?php
 2
 3declare(strict_types=1);
 4
 5namespace Phplrt\Contracts\Lexer;
 6
 7/**
 8 * A channel that is defined outside of the basic set of the token channels.
 9 */
10readonly class UserDefinedChannel implements ChannelInterface
11{
12    public function __construct(
13        /**
14         * The name of the channel.
15         *
16         * @var non-empty-string
17         */
18        public string $name,
19    ) {}
20}

3.6 `Phplrt\Contracts\Lexer\Exception\LexerExceptionInterface`

 1<?php
 2
 3declare(strict_types=1);
 4
 5namespace Phplrt\Contracts\Lexer\Exception;
 6
 7/**
 8 * An error of the lexical analysis.
 9 *
10 * Every exception thrown by a lexer MUST implement this interface.
11 */
12interface LexerExceptionInterface extends \Throwable {}

3.7 `Phplrt\Contracts\Lexer\Exception\RuntimeExceptionInterface`

 1<?php
 2
 3declare(strict_types=1);
 4
 5namespace Phplrt\Contracts\Lexer\Exception;
 6
 7use Phplrt\Contracts\Lexer\TokenInterface;
 8use Phplrt\Contracts\Source\ReadableInterface;
 9
10/**
11 * An error that occurs after the lexical analysis has been started and
12 * indicates a problem in the analyzed source.
13 */
14interface RuntimeExceptionInterface extends LexerExceptionInterface
15{
16    /**
17     * The source the error occurred in.
18     */
19    public ReadableInterface $source {
20        get;
21    }
22
23    /**
24     * The token the error occurred on.
25     */
26    public TokenInterface $token {
27        get;
28    }
29}

4. See Also

Usage - the phplrt lexer itself.
Tokens and Channels - what a lexer does with channels.
Parser Contracts - what consumes these tokens.

Lexer Contracts

1. Specification

1.1 Lexer

1.2 Tokens

1.3 Channels

1.4 Errors

2. Package

3. Interfaces

3.1 Phplrt\Contracts\Lexer\LexerInterface

3.2 Phplrt\Contracts\Lexer\TokenInterface

3.3 Phplrt\Contracts\Lexer\ChannelInterface

3.4 Phplrt\Contracts\Lexer\Channel

3.5 Phplrt\Contracts\Lexer\UserDefinedChannel

3.6 Phplrt\Contracts\Lexer\Exception\LexerExceptionInterface

3.7 Phplrt\Contracts\Lexer\Exception\RuntimeExceptionInterface