阅读中文版本

The Context Window

A model's context window bounds how many tokens it can consider at once, a hard limit that governs truncation, cost, and how much retrieved text you can supply.

Definition

The context window is the maximum number of tokens a language model can attend to in a single request, comprising the system prompt, conversation history, retrieved context, and the generated completion.

It is a primary constraint when designing prompts, retrieval pipelines, and multi-turn conversations. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.

When to use it

A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.

How it works

Every part of a request competes for the same budget, so a long retrieved document can push out conversation history or instructions without warning.

Attention degrades with position, and content in the middle of a very long input is often used less reliably, which favours putting key facts near the edges.

Trade-offs and alternatives

Managing the window means selecting, summarising, or dropping content deliberately rather than letting a naive concatenation overflow. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.

Common misconceptions

A persistent misconception is that text within the window is always used equally well. Another is that a larger window eliminates the need for retrieval or summarisation. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.

Anti-patterns to avoid

Two recurring anti-patterns are to filling the window with low-value context and crowding out instructions and to never truncating history and silently dropping the system prompt. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include Prompt Injection, Semantic Caching, Document Chunking Strategies.