CertScroll

Tokenization

Splitting text into the subword units a model actually processes, using a fixed vocabulary learned from training data rather than whitespace or dictionary words. A word outside that vocabulary gets split into smaller known pieces instead of failing outright. Every piece counts against the context window, which is why token count and word count are never the same number.