The Ultimate AI Security Checklist for Developers and Startups

This is the checklist we would use before putting an AI feature in front of real users. It is ordered by blast radius: the items at the top prevent the worst outcomes even if detection fails.

Identity and tenancy

• ☐Every AI endpoint requires an authenticated session or a scoped machine key.
• ☐Tenant and user are derived server-side, never from the prompt or request body.
• ☐Every retrieval query, cache key and tool credential is tenant-scoped.
• ☐A second account cannot read the first account's objects by guessing an id.
• ☐Admin and support access is explicit, logged and reviewed.

Prompts and model calls

• ☐No secrets, internal URLs or credentials in the system prompt.
• ☐The prompt labels which content is data and which is instruction.
• ☐User input, retrieved documents and tool results are screened for injection.
• ☐Request size and conversation length are capped.
• ☐The model and version are pinned and recorded with each request.
• ☐A known-good fallback exists when the model provider is unavailable.

Retrieval

• ☐Only documents the user may see are indexed for that tenant.
• ☐Permissions are checked at query time, not only at ingestion.
• ☐Retrieved chunks are treated as untrusted content.
• ☐Sensitive collections are excluded or separately governed.
• ☐Re-indexing happens when permissions change.

Agents and tools

• ☐Each tool has its own scoped, short-lived credential.
• ☐Tool arguments are validated against a schema.
• ☐Destinations, recipients, tables and commands are allowlisted where possible.
• ☐Destructive and financial actions require human approval or proof of intent.
• ☐Steps, tokens, spend and wall-clock time are capped per run.
• ☐MCP servers are inventoried, version-pinned and audited.
• ☐The agent cannot install packages or open a shell unless that is the product.

Output handling

• ☐Model output is treated as untrusted input by every downstream system.
• ☐HTML and markdown are sanitized.
• ☐Generated URLs and images are restricted to allowed schemes and hosts.
• ☐SQL and shell use parameterization, never string interpolation.
• ☐DLP checks secrets, credentials and personal data before the response leaves.
• ☐Generated content is labelled where users could mistake it for verified fact.

Abuse and cost

• ☐Rate limits exist per IP, per user and per tenant.
• ☐Expensive endpoints have their own budgets.
• ☐Per-session token and cost ceilings are enforced.
• ☐Loops and repeated retries are detected.
• ☐Alerts fire before the invoice, not after.

Evidence and response

• ☐Every verdict is logged with input hash, controls, policy version and action.
• ☐The log is append-only or tamper-evident.
• ☐Sensitive content is not retained longer than necessary.
• ☐There is a tested way to disable the feature or the model quickly.
• ☐An incident runbook names who is called and what is preserved.
• ☐A tabletop exercise has been run with a realistic injection scenario.

Supply chain

• ☐Model artefacts and adapters are hashed and verified.
• ☐Inference containers and SDKs are pinned and scanned.
• ☐Model updates go through review and can be rolled back.
• ☐Third-party tools and MCP servers have an owner and a review date.

What to do first

If you can only do five things this week:

1. Remove standing credentials from agents and scope every tool.

2. Put a screening gateway in front of prompts and responses.

3. Treat model output as hostile at every downstream boundary.

4. Add per-session spend caps.

5. Log every verdict in a store you can trust.

That sequence reduces blast radius before it tries to improve classification. Detection will always miss something; the boundaries are what keep a miss from becoming an incident.