A security review checklist for enterprise AI systems
What to ask before an AI system reaches production. Written to be handed to a security team that has not reviewed one before.
Most security teams reviewing their first AI system reach for the application security checklist they already have. It covers perhaps half of what matters, and the half it misses is where the interesting failures live.
This checklist covers the AI-specific surface. It assumes your existing controls — network, identity, patching, secrets, logging — are already applied. Use it alongside them, not instead.
Take it and use it. No attribution required. If it saves a review cycle it has done its job.
1. Data boundary
The first question, and the one that eliminates the most architectures.
- Where does inference physically execute, and who operates that infrastructure?
- Does any prompt, retrieved content, or output leave your network boundary? Including to logging, analytics, or evaluation services?
- If a third-party API is used: what are the retention terms, are they configured, and has that configuration been verified rather than assumed?
- Is the data used for training or model improvement? Is that contractual or merely stated in documentation?
- Where do embeddings, indexes, caches, and logs persist? Embeddings derived from sensitive text are sensitive.
- Does observability tooling emit prompts or completions to an external service?
That last one catches people. Several popular LLM observability platforms are hosted, and teams wire them up without registering that full prompt content is now leaving the perimeter.
2. Retrieval and entitlements
The most common serious finding we see, and rarely caught by generic review.
- Are entitlements enforced at query time, against the same source of truth the document system uses?
- Or are they applied at index time — and therefore stale the moment permissions change?
- Is filtering applied before retrieval, or after? Filtering after means the retriever already read across the boundary.
- Can a user retrieve through the AI what they could not open directly? Test this explicitly with two accounts of differing privilege.
- Does caching respect entitlements, or can a cached result cross users?
- Can the system aggregate across documents to produce disclosure that no single document permission would have allowed?
The aggregation question is the subtle one. A user may legitimately have access to a hundred documents individually while the synthesis of all hundred constitutes information they were never meant to hold. Traditional access control has no concept of this. Decide deliberately whether it matters in your context.
3. Prompt injection
Direct injection — a user typing adversarial instructions — is the easy case. Indirect injection, where instructions arrive inside content the system retrieves, is the hard one.
- Is all retrieved content treated as untrusted input rather than as instruction?
- Can content in the corpus be authored or influenced by someone outside the trust boundary? Email, uploaded files, scraped web content, ticket bodies?
- What is the blast radius if an injection succeeds — what tools, data, or actions become reachable?
- Are defenses architectural, or do they live in the system prompt? Prompt-only defenses are not defenses.
- Has injection been tested against the deployed system, not the model in isolation?
A useful framing for reviewers: assume injection succeeds. Now ask what the attacker can do. If the answer is "read one document the user could already read", the risk is manageable. If it is "send email as the user", it is not.
4. Agent authority
Where an AI system can take actions, the security question shifts from data to authority.
- What tools can the agent invoke, and with whose credentials?
- Is tool access scoped per task, or does the agent hold the union of all permissions it might ever need?
- Which actions are consequential — send, delete, transact, escalate, modify — and do those require human approval?
- Is there a rate or spend limit on tool invocation?
- Can the agent invoke itself recursively, or chain into another agent with broader permissions?
- Is every tool call logged with the input that triggered it?
5. Output handling
Model output is untrusted input to whatever consumes it next. This is a classic vulnerability class wearing new clothes.
- Is output rendered as HTML or markdown anywhere? Can it inject script, or a link that exfiltrates via a query parameter?
- Is output ever used to construct a query, a command, or a file path?
- Can output containing an image reference cause an outbound request to an attacker-controlled host?
- Are citations verified to point at documents the user may actually access?
The markdown image exfiltration path is worth checking specifically: a model induced to emit an image tag pointing at an external host, with sensitive content encoded in the URL, will cause the user's browser to make that request when the response renders.
6. Supply chain
- Where did the model weights come from, and is their integrity verified against a published checksum or signature?
- For fine-tuned or community models: what is the provenance of both base model and training data?
- Are inference server, tokenizer, and embedding dependencies pinned and scanned?
- For air-gapped environments: is everything mirrored internally, and are transitive dependencies pinned?
7. Evaluation and monitoring
Security review of an AI system should ask about quality, because a confidently wrong answer in a clinical or financial context is a safety issue rather than a product defect.
- Is there an evaluation set built from real data, with agreed acceptance thresholds?
- Does evaluation run continuously against production traffic, or only before launch?
- Is there alerting on quality regression, not just on latency and errors?
- Is there a defined process for model version changes, including deprecation?
- Can you reconstruct what the system saw and produced for any given request?
8. Audit evidence
Someone will eventually ask what the system did and on what basis. Design for that question rather than reconstructing an answer under pressure.
- Are inputs, retrieved context, model version, parameters, and output all logged?
- Is the log retention aligned to your records schedule?
- Do logs themselves contain sensitive content, and are they protected accordingly?
- Can you demonstrate to an auditor which controls enforce the data boundary?
- Is there a documented human oversight point for consequential decisions?
How to use this
Run it as a design review during architecture, not as a gate before launch. Every item above is cheap to address while the design is fluid and expensive once the system is built.
If you are reviewing a system that already exists, work top-down. Data boundary and retrieval entitlements are where the findings that stop deployment usually are — and they are also the two that are hardest to retrofit.
The mapping to formal frameworks — NIST AI RMF, ISO/IEC 42001, and the OWASP Top 10 for LLM Applications — is worth doing once, so findings land in a process your organization already runs rather than in a document nobody owns.
Related
AI for Financial Services
AI for Financial ServicesStuck at one of these?
The assessment diagnoses which constraint is actually blocking you, and produces a roadmap you own — executable by us, by another firm, or by your own team.
Direct response from an engineer. Typically within one business day.