阅读中文版本

Prompt Injection

Prompt injection embeds instructions inside untrusted content so the model follows the attacker instead of the developer, and it cannot be fully solved by prompting alone.

Definition

Prompt injection is an attack in which adversarial text, often placed in retrieved documents or user content, causes a language model to disregard its original instructions or leak information.

It affects any application that lets a model read content the developer does not control, including chat, summarisation, and retrieval-augmented generation. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.

When to use it

A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.

How it works

Because the model treats all text in the window as potential instructions, content with no trusted provenance can hijack behaviour, and there is no reliable text-only filter.

Defence in depth applies: separate trusted instructions from untrusted data, constrain the actions a model can take, and validate outputs before acting.

Trade-offs and alternatives

Any tool the model can call is a real capability, so least privilege and human confirmation for consequential actions limit the blast radius. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.

Common misconceptions

A persistent misconception is that a strong system prompt prevents injection. Another is that injection only matters for chatbots that act on instructions. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.

Anti-patterns to avoid

Two recurring anti-patterns are to concatenating retrieved text into the same turn as system instructions and to granting a model tools without limiting their effects. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include Semantic Caching, Document Chunking Strategies, AI Guardrails and Output Validation.