Module types 

Sub-scanners for each literal and identifier kind.

Each sub-module is an impl Tokenizer block that handles one token class:

• character_literal - single-quoted 'x' characters
• identifier - keywords and user-defined names
• number_literal - integers and floats
• string_literal - double-quoted "…" strings

Modules

character_literal 🔒

Single-quoted character literal scanner.

identifier 🔒

Identifier and keyword scanner.

number_literal 🔒

Integer and float literal scanner.

string_literal 🔒

Double-quoted string literal scanner.