Security review for LLM features, the questions that matter
Your security team knows how to review software. LLM features add five genuinely new questions. Here they are, with the answers that should pass.
Security teams keep asking me, a little wearily, what is actually new about reviewing LLM features. Most of the review is the same software review they already run well. Five questions are genuinely new, and a review that asks them covers most of what the OWASP GenAI Top 10 catalogs at length.
What can the model be talked into doing? Prompt injection is not a bug to patch; it is a property to design around. Anything the model can do, a sufficiently hostile input will eventually make it do. The passing answer inventories the model's capabilities (tools, data access, outbound calls) and shows each one bounded by infrastructure, not by instructions in the system prompt.
Where does pasted data go? Which vendor, which retention terms, which region, which training clauses. The passing answer is a data flow diagram that includes the model provider as a processor, not a footnote.
Can retrieval leak across permissions? If the assistant answers from documents, it must inherit the document ACLs. The failing answer, common in demos, is one big index with the permissions checked nowhere.
What do the logs capture, and who may read them? Prompts and outputs are logs that contain whatever users typed, which means the logging pipeline just became a sensitive datastore. Retention and access rules apply to it on day one, not after the first incident.
What happens when the output is wrong? Not whether; when. The passing answer names the human checkpoint for consequential actions and the blast radius of an unchecked mistake.
How the review runs in practice. Add the five questions to the existing design-review template and require written answers before the feature reaches staging, which is earlier than security usually gets invited and precisely the point. The answers should be a page total. If a team cannot fill that page, the feature is not blocked by security; it is blocked by not being designed yet, and the review just said so out loud. In audits I read these pages before anything else, because a team's five answers predict the rest of the engagement: crisp answers mean the boring controls exist too, and adjectives mean we are about to find an index with everyone's documents in it. (We usually are.)
The two answers that fail most often. Retrieval ACLs and logs. The retrieval failure happens because the demo was built on a test corpus where permissions did not matter, and the production corpus arrived later with the permissions assumed to be someone else's layer. Budget the ACL inheritance work when the index is designed, not when the first "why can I see the CEO's notes" ticket lands. The logging failure is more innocent: engineers log generously because debugging LLM features is hard, and nobody reclassifies the log store after it starts holding pasted contracts. One meeting with whoever owns data classification fixes it, provided the meeting happens before the incident instead of inside it.
What to do with vendors' features, since most LLM surface arrives bought rather than built. The same five questions apply, answered from the vendor's documentation and contract instead of your diagrams, and the gaps become procurement questions rather than engineering tickets. A vendor who cannot say where pasted data goes has answered question two in the way that ends meetings early. Keep the answers on file per product; when the renewal or the incident arrives, that one page is the difference between a decision and a scramble.
A note for the security team's side of the table. The wrong posture is treating every LLM feature as novel and terrifying, because teams respond by routing around review entirely, and the shadow inventory grows a new wing. The right posture is the one your best reviewers already use for everything else: same rigor, same templates, five new questions, answers in writing. Novelty is the enemy of consistency, and consistency is most of what security review is for.
The reviews also age differently than standard ones, which is worth scheduling for. A normal feature's security posture decays with its code changes; an LLM feature's posture also decays when the model underneath updates, when the corpus grows a new document type, and when users invent workflows the design never met. Put a light re-review on the calendar at six months, scoped to the same five questions and an hour. It is the cheapest security work in the entire program, and it is where you find the index that quietly started ingesting the HR drive in March.
None of this requires a new security organization. It requires adding five questions to a checklist that already exists, and declining to ship features whose answers are adjectives.