Module lexer 

The lexer - converts raw source text into a flat Vec<Token>.

Pipeline position

source -> Lexer -> [Token] -> Parser -> [Statement] -> Checker -> Evaluator

Module layout

• tokenizer - the [Tokenizer] struct and its main scanning loop
• tokentypes - the [Token] and [TokenType] definitions
• scanner - top-level scan driver called by the pipeline
• types - sub-scanners for each literal kind (string, char, number, identifier)
• utils - shared cursor helpers used across the sub-scanners

Modules

scanner 🔒

Top-level scan driver.

tokenizer

Tokenizer struct and the main character-by-character scan loop.

tokentypes

Token and TokenType - the complete vocabulary of the lexer.

types 🔒

Sub-scanners for each literal and identifier kind.

utils 🔒

Shared cursor helpers for the lexer sub-scanners.