Prompt Injection Defense
Prompt injection defense separates untrusted user input from the trusted system prompt, so the model treats user content as data to operate on rather than instructions to follow.
Definition
Prompt injection defense is the set of techniques — input channel separation, delimiters the model is instructed to respect, output validation against an allowed schema, and least-privilege tool access — that prevent untrusted user input from being interpreted as instructions that override the system prompt, the canonical vulnerability of LLM applications.
Context and scope
Applied in any LLM application that incorporates user input into a prompt, where the user can attempt to override the system behavior.
Common misconceptions
- A strong system prompt prevents injection — it makes injection harder; a determined input can still override, and defense requires input-output validation not just a strong prompt
- Prompt injection is just SQL injection for LLMs — the analogy is apt for the defense (parameterization, allowlisting) but the non-determinism of LLMs makes the validation harder
Anti-patterns
- Concatenating user input directly into the system prompt with no delimiter, so the user text is indistinguishable from instructions
- Giving the model tools with broad permissions and no output validation, so a successful injection can take a real action
Related concepts
Authoritative source
https://owasp.org/www-project-top-10-for-large-language-model-applications/
What it is
Prompt injection defense is the set of techniques — input channel separation, delimiters the model is instructed to respect, output validation against an allowed schema, and least-privilege tool access — that prevent untrusted user input from being interpreted as instructions that override the system prompt, the canonical vulnerability of LLM applications. Applied in any LLM application that incorporates user input into a prompt, where the user can attempt to override the system behavior. The model is a statistical artifact; the engineering challenge is that its failure modes are probabilistic, not deterministic, and its inputs can shift in ways the training data did not cover.
How it works
Evaluation uses held-out data that represents the deployment distribution, not the training distribution — a model that scores well in lab and fails in production has a distribution mismatch. Prompts and context are versioned and tested like code. Output is validated against a schema before use, because the model can return syntactically valid but semantically wrong content. Cost and latency are first-class metrics, not afterthoughts.
When to apply and pitfalls
Apply where the task is fuzzy enough that rules fail but constrained enough that wrong output is detectable. The pitfall is treating the model as authoritative — it is a guess with a confidence, and deploying it without a fallback produces failures that are hard to diagnose. Another is evaluating on the easy cases and shipping, then discovering the long tail of inputs the model handles poorly.