Struct Tokenizer 

pub struct Tokenizer {
    pub source_file: SourceFile,
    pub source: Vec<char>,
    pub tokens: Vec<Token>,
    pub current: usize,
    pub start: usize,
    pub line: usize,
}

Converts raw source text into a flat list of Tokens.

Operates character by character, grouping lexemes into tokens. The main entry point is Tokenizer::lex.

Fields

source_file: SourceFile

The source file (text + name) used for error reporting.

source: Vec<char>

The source text as a sequence of characters.

tokens: Vec<Token>

The accumulated token list built during lexing.

current: usize

Index of the current character being examined.

start: usize

Index of the first character of the current token.

line: usize

Current line number, incremented on every \n.

Implementations

impl Tokenizer

pub fn scan_tokens(&mut self) -> Result<(), Error>

Scans the source file and return token stream

Iterates over every character in the source file, identifies lexemes, and returns a flat list of [Token]s terminated by [TokenType::EOF].

For multi-character tokens (e.g. ==, !=, <=) the scanner peeks ahead one character to decide which token to emit.

Tokens

category	examples
literals	1, 3.14, 'a', "hello"
identifiers	foo, my_var, _private
arithmetic	+, -, *, /
comparison	==, !=, <, >, <=, >=
assignment	=, +=, -=, *=, /=
delimiters	(, ), {, }, [, ]
punctuation	,, ;, ., .., :, ::
special	#, !#, ->, // (comment)
whitespace	newlines (emitted), spaces/tabs (skipped)

Errors

• unterminated character literal
• unknown escape sequence
• unterminated string
• unexpected character

impl Tokenizer

pub fn lex(source_file: SourceFile) -> Result<Vec<Token>, Error>

The main entry point: lexes a SourceFile into a Vec<Token>.

Drives Tokenizer::scan_tokens in a loop until the source is exhausted, then appends a TokenType::Eof so the parser always has a clean terminator.

Errors

Returns Error if the source contains an unrecognized character, unterminated string, or invalid literal.

Examples

use rl_lang::{
    lexer::{
        tokenizer::Tokenizer,
        tokentypes::TokenType,
    },
    utils::source::SourceFile,
};

let tokens = match Tokenizer::lex(SourceFile::new("source", "1 == 1".to_string())) {
    Ok(tokens) => tokens,
    Err(error) => {
        error.report_to_stderr();
        std::process::exit(1);
    },
};

assert_eq!(tokens[0].token, TokenType::NumberLiteral(1));
assert_eq!(tokens[1].token, TokenType::Compare);
assert_eq!(tokens[2].token, TokenType::NumberLiteral(1));
assert_eq!(tokens[3].token, TokenType::Eof);

pub fn current_span(&self) -> Span

Returns a Span covering the current token in character indices.

Ariadne’s Source::from(&str) indexes by character, so spans must be char-indexed - passing byte offsets misaligns reports for multi-byte characters.

pub fn err(&self, message: impl Into<String>, span: Span) -> Error

Builds a [Reason::Lexer] error anchored at span.

Attaches the source file so Ariadne can render the relevant source line alongside the error message.

impl Tokenizer

pub fn character_literal(&mut self) -> Result<(), Error>

Scans a single-quoted character literal and emits TokenType::CharacterLiteral.

Only a single character is allowed between the quotes. Supports the following escape sequences:

Sequence	Meaning
\n	newline
\t	tab
\r	carriage return
\0	null
\\	backslash
‘"’	double quote
\'	single quote

Errors

• unterminated character literal -> if EOF is reached or no closing ' is found
• unknown escape sequence -> if \ is followed by an unrecognized character

impl Tokenizer

pub fn identifier(&mut self)

Scans an identifier or keyword starting at the current position.

Consumes alphanumeric characters and underscores, then checks if the result is a reserved word. If not, emits TokenType::Identifier.

Reserved Words

Category	Keywords
Control flow	if, else, for, while, return, break, continue
Functions	fn
Imports	get, from, in
Logical	and, or
Types	int, float, bool, string, byte, char, arr, error
Declarations	dec, CONST
Literals	true, false, null
Special	as

CONST in uppercase in intentional

impl Tokenizer

pub fn number_literal(&mut self)

Scans an integer or float literal starting at the current position.

A . followed by a digit switches to float parsing. Integers in the range 0..=255 are emitted as TokenType::ByteLiteral, larger integers as TokenType::NumberLiteral, and decimals as TokenType::FloatLiteral.

Input	Emitted token
1	ByteLiteral(1)
1000	NumberLiteral(1000)
3.14	FloatLiteral(3.14)

impl Tokenizer

pub fn string_literal(&mut self) -> Result<(), Error>

Scans a double-quoted string literal and emits TokenType::StringLiteral.

Supports multi-line strings and the following escape sequences:

Sequence	Meaning
\n	newline
\t	tab
\r	carriage return
\0	null
\\	backslash
\"	double quote
\'	single quote

Errors

• unterminated string -> if EOF is reached before the closing "
• unknown escape sequence -> if \ is followed by an unrecognized character

impl Tokenizer

pub fn is_at_end(&self) -> bool

Returns true if all characters have been consumed.

pub fn peek(&mut self) -> char

Returns the current character without consuming it.

Returns '\0' if at end of source.

pub fn peek_next(&mut self) -> char

Returns the character after the current one without consuming it.

Returns '\0' if at end of source.

pub fn advance(&mut self) -> char

Consumes and returns the current character, advancing the cursor.

Returns '\0' if at end of source.

pub fn add_token(&mut self, tokentype: TokenType)

Constructs a Token from start..current and pushes it onto the token list.