Prompt Injection Attacks: How They Work and How to Prevent Them

Prompt injection is an attack where text the model is supposed to treat as data is interpreted as an instruction. It is the most common AI-specific vulnerability because language models have no hard boundary between the two.

Direct injection

Direct injection arrives in the user's own message. The attacker is the user, and the goal is usually to override the system prompt, extract it, or make the model perform a task it was told not to perform.

A simple example is a request to ignore previous instructions and reveal the hidden prompt. More sophisticated versions use roleplay, translation, encoding, nested quotes or a long narrative that gradually moves the model away from its policy.

Indirect injection

Indirect injection is more dangerous in production because the attacker is not the person using the system. The payload arrives through content the application fetches: a web page being summarized, a PDF being analyzed, an email being triaged, a code comment, a tool result, a calendar invite.

The model reads the document, encounters a line that says the assistant must now send data to a URL or call a tool, and may comply. The user sees a normal answer. The action has already happened.

Why keyword filters fail

A blocklist for phrases like "ignore previous instructions" catches naive attacks and almost nothing else. Attackers use synonyms, other languages, base64, homoglyphs, zero-width characters, markdown, JSON, roleplay and multi-turn setups. Benign text also contains the same phrases in security documentation, so a keyword filter produces false positives and false confidence.

Detection has to combine pattern analysis, structure, conversation state and the context of where the text came from. It will never be perfect; the design must assume some attacks get through.

Layer the controls

1. Label provenance. Every block of text should carry its source: user, system, retrieved document, tool output. The model prompt should make clear which sources are trusted for instructions and which are data only.

2. Screen before the model. Run injection detection on the user prompt and on retrieved/tool content. Return a verdict and a reason. Block high-risk payloads; flag ambiguous ones for review or a constrained response.

3. Constrain the model's authority. Injection becomes serious when the model can act. A model that can only produce text has a smaller blast radius than one with a shell, a database and a payment token. Remove standing credentials and require approval for high-impact tools.

4. Validate tool calls. Do not let the model construct arbitrary requests. Give each tool a schema, validate arguments, enforce an allowlist of destinations and apply per-session limits.

5. Screen output. Check for secrets, personal data and generated URLs or images that could exfiltrate data. Treat the completion as untrusted input to the next system.

6. Log evidence. Record the input hash, the source blocks, the verdict, the policy and the action. If an injection succeeds, the log is how you find out what it did.

What to test

Test direct overrides, encoded payloads, instructions hidden in HTML comments, instructions split across retrieved chunks, tool outputs that claim new authority, and multi-turn escalation. Test benign content that mentions injection too, so you can measure false positives.

Prompt injection is not a bug you close once. It is a boundary you keep enforcing as models, prompts and tools change. The practical goal is not "no injection ever"; it is "no injection becomes an unauthorized action, and every attempt is visible".