Top 10 Security Threats Facing AI Applications

Most AI incidents are not exotic. They combine a familiar web vulnerability with a model that was given more trust than the data deserved. These are the ten threats we see most often, and the control that actually reduces each one.

1. Prompt injection

An attacker puts instructions in text the model reads as data: a user prompt, a retrieved document, a web page, a tool result. The model follows them. Direct injection arrives in the prompt; indirect injection arrives through content the system fetched. The control is screening both the prompt and every retrieved or tool-returned block before the model acts on it.

2. Jailbreaks and policy bypass

A jailbreak persuades the model to ignore its own safety policy through roleplay, encoding, translation or a long multi-turn story. It is different from injection: injection hijacks the task, a jailbreak removes the guardrail. Detection needs to read intent and structure across the conversation, not match a single phrase.

3. Sensitive data disclosure

Models repeat what they have seen. A retrieval system that indexes too much can surface another customer's record. A prompt can ask for secrets, system instructions or internal identifiers. The controls are retrieval scoping, output DLP, redaction, and testing that the model cannot be talked into revealing context it should not have.

4. Data exfiltration through outputs

This is the outbound version: the model is convinced to place data in a URL, an image tag, a markdown link or a tool call that leaves the environment. Blocking outbound destinations and inspecting generated URLs and images matters more than inspecting the prompt alone.

5. Insecure tool and plugin use

Tools turn a text problem into an action problem. A plugin with a broad OAuth token, a shell tool with no allowlist, or a database tool that accepts raw SQL gives the model real authority. Use least privilege, per-tool schemas, argument validation and human approval for destructive actions.

6. Excessive agency

An agent that can plan, browse, write and pay without checkpoints will eventually do something expensive. Excessive agency is a design flaw. Bound the goal, the number of steps, the spend and the set of reachable systems. Require proof of intent for high-impact actions.

7. Model and supply-chain compromise

A model file, adapter, container or dependency can be swapped. The system then behaves normally in tests and maliciously in production. Verify provenance, pin hashes, scan artefacts and treat model updates like code deployments.

8. Denial of wallet

Recursive prompts, runaway loops or abusive clients consume tokens until the bill or a quota stops them. Rate limits per user are not enough; add per-session token and cost ceilings, loop detection and alerts before the invoice arrives.

9. Insecure output handling

The model's output is untrusted input to the next system. If it is rendered as HTML, executed as code, passed to a shell or used to build a query, it can carry XSS, command injection or SQL injection. Encode, validate and sandbox downstream use exactly as you would for any user input.

10. Weak observability and evidence

If you cannot reconstruct what the model saw, which controls ran and what action was taken, you cannot investigate an incident or defend a decision. Log input hashes, verdicts, policy versions and actions in an append-only, tamper-evident store. Keep enough context to explain the decision without retaining raw traffic longer than necessary.

How the threats combine

A single incident often chains several of these. An indirect injection in a retrieved page convinces an agent to call a tool with a broad token, which exfiltrates data through a generated URL, and the only evidence is a normal-looking completion. Defence in depth is not a slogan here; each boundary needs its own check.

Start with the controls that reduce blast radius: least privilege, output handling, spend ceilings and evidence. Then improve detection. A perfect classifier with an all-powerful agent is still an incident waiting to happen.