概念概述
分词与子词单元（Tokenization and Subword Units）指的是：分词把文本拆成模型处理的子词单元，其选择影响上下文上限、成本以及对罕见词或非拉丁文本的处理。
从更严格的定义看，Tokenization is the transformation of raw text into a sequence of tokens, often subword units, that a language model can map to embeddings and process within its fixed vocabulary. 该概念在工程实践中的价值，在于它能帮助团队用统一的语言描述一个具体的质量属性或设计约束，从而在评审与协作中减少歧义。
为什么需要它
It matters whenever counting context length, estimating cost, or reasoning about why a model struggles with a particular script. 它与 The Context Window、Prompt Injection、Semantic Caching 等概念经常一起出现，用于解决相近的工程问题。理解它的适用场景，有助于在多种备选方案之间做出有依据的取舍，而不是凭直觉选择技术方案。
常见误解
- a token equals a word in every language
- token counts are identical across all models
需要避免的反模式
- budgeting context by character count instead of tokens
- assuming text in every script tokenises as efficiently as English
实践要点
- Subword schemes let a fixed vocabulary represent rare words by composing pieces, which allows the model to handle novel terms without an unknown token.
- Scripts and languages differ in tokens per character, so the same sentence can consume very different context and cost depending on the language.
- Context limits and pricing are expressed in tokens, so any truncation or cost estimation must use the specific model’s tokenizer.