Top-level scan driver.
Exposes the single entry point that the pipeline calls to turn a source string into a token stream. Delegates all real work to Tokenizer.
Tokenizer