Tokenization and Subword Units
Tokenization splits text into the subword units a model processes, and its choices shape context limits, cost, and handling of rare or non-Latin text.
Definition
Tokenization is the transformation of raw text into a sequence of tokens, often subword units, that a language model can map to embeddings and process within its fixed vocabulary.
Context and scope
It matters whenever counting context length, estimating cost, or reasoning about why a model struggles with a particular script.
Common misconceptions
- a token equals a word in every language
- token counts are identical across all models
Anti-patterns
- budgeting context by character count instead of tokens
- assuming text in every script tokenises as efficiently as English
Related concepts
Authoritative source
Definition
Tokenization is the transformation of raw text into a sequence of tokens, often subword units, that a language model can map to embeddings and process within its fixed vocabulary.
It matters whenever counting context length, estimating cost, or reasoning about why a model struggles with a particular script. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.
When to use it
A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.
How it works
Subword schemes let a fixed vocabulary represent rare words by composing pieces, which allows the model to handle novel terms without an unknown token.
Scripts and languages differ in tokens per character, so the same sentence can consume very different context and cost depending on the language.
Trade-offs and alternatives
Context limits and pricing are expressed in tokens, so any truncation or cost estimation must use the specific model’s tokenizer. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.
Common misconceptions
A persistent misconception is that a token equals a word in every language. Another is that token counts are identical across all models. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.
Anti-patterns to avoid
Two recurring anti-patterns are to budgeting context by character count instead of tokens and to assuming text in every script tokenises as efficiently as English. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include The Context Window, Prompt Injection, Semantic Caching.