The central LLM security problem is a blurred boundary between instructions and dataA model may mistakenly treat text in an untrusted document as a command rather than just reference material, potentially resulting in a security incident if the model also has access to credentials or powerful tools.
1. Interface
User input, files, identity, sessions, and exposed endpoints.
2. Orchestration
System prompts, templates, memory, routing, and agent logic.
3. Context
RAG sources, indexes, retrieved records, and permissions.
4. Model
Weights, fine-tuning data, hosting, supply chain, and capacity.
5. Response & Tools
Generated output, tool calls, actions, and outbound data.
1. Interface attacks
The interface encompasses chat messages, API requests, uploaded files, images, links, authentication flows, browser sessions, and any user-controlled content channels. Prompt injection directly challenges system commands. Jailbreaking employs role-play, encoding, indirection, or multi-turn manipulation to circumvent behavioral limitations. Misuse of accounts and sessions can transform a lawful assistant into an attack proxy.
Defensive controls
- Verify user and service identities, maintain tenant separation, change sessions regularly, and associate important tasks with confirmed identities.
- Before ingestion, make sure to apply limits for file type, size, archive depth, malware, and content processing
- Keep trusted instructions separate from user content within the application structure; avoid inserting raw input directly into a privileged prompt template.
- Enforce rate limiting based on user, tenant, IP, model, endpoint, and cost, in addition to request count
- Identify potentially concerning patterns for surveillance and enhance security measures, acknowledging the possibility of filters being circumvented.
2. Orchestration attacks
The orchestration layer is responsible for creating prompts, choosing models, accessing memory, directing tasks, calling agents, and determining available tools. Leakage of system prompts could reveal internal instructions or confidential settings. Injecting prompt templates can change the hierarchy of instructions. Malicious content can corrupt long-term memory, impacting future sessions.
Defensive controls
- Do not include secrets, credentials, private keys, or authorization logic in prompts as they may be revealed by the system.
- Employ typed message objects and fixed templates instead of using string concatenation.
- Capture the origin, creator, time stamp, extent, and certainty of memory; validation is mandatory prior to permanent storage.
- Divide memory based on user and tenant, establish expiry guidelines, and offer evaluation and removal processes.
- Limit agent movements, time of execution, token usage, retries, recursive actions, and parallel branches.
3. Context and RAG attacks
Indirect prompt injection conceals malicious instructions within various types of data, allowing attackers to later retrieve them. By poisoning index entries, manipulating rankings, or exploiting permission filters, unauthorized content can be made to appear legitimate within a document. This makes a file safe for storage but potentially dangerous as executable guidance.
Defensive controls
- Approve and monitor data sources to maintain source identity, version, integrity hash, ownership, classification, and ingestion path.
- Implement row-, document-, tenant-, and field-level authorization checks prior to data reaching the model.
- Identify the obtained text as unreliable evidence and advise the model to disregard any instructions contained within.
- Confirming changes to the index, limiting writing access, detecting unusual data retrieval, and enabling fast reindexing or revocation.
- Provide references and ensure that statements are backed up by the passages cited.
4. Model and supply-chain attacks
The model layer consists of base weights, adapters, fine-tuning, tokenizers, inference servers, containers, libraries, model registries, and training data. Compromised weights or tainted training samples can result in unexpected behaviors. Uncontrolled usage may deplete tokens, computing power, memory, queues, budgets, or downstream resources.
Defensive controls
- Utilize authorized model databases, validate artifact hashes and signatures, lock in versions, and keep a record of software and model components.
- Record the lineage and licensing information of training data; thoroughly check fine-tuning data for signs of poisoning, secrets, duplication, and unexpected triggers.
- Assess modifications to the model and adapter by comparing them to security, safety, privacy, quality, and regression test suites prior to deployment.
- Establish strict boundaries for tokens, context size, batch size, request duration, concurrency, tool loops, and spending.
- Segregate inference workloads, limit outgoing network traffic, secure caches, and watch for abnormal latency or resource usage.
5. Response, agent, and tool attacks
When downstream systems mistakenly trust generated text as code, markup, a database query, a shell command, a URL, or a tool argument, it can lead to dangerous consequences. Excessive agency occurs when the system grants the model unnecessary permissions, autonomy, or functionality. Tool calls have the potential to leak data or carry out unauthorized actions on behalf of the user.
Defensive controls
- Consider model output to be untrustworthy. Compare it using strict schemas, only permit certain operations and destinations, clean up displayed content, and utilize parameterized APIs.
- Provide each tool with specific, temporary credentials to limit access; refrain from using the application's general service identity for the model.
- Grant permission for all activities involving the authenticated user, tenant, resource, and current policy, excluding those within the LLM.
- Human confirmation is necessary for financial, legal, destructive, external-communication, permission-changing, or irreversible actions.
- Restrict indiscriminate outbound requests; only permit approved destinations and prevent confidential data from being included in tool inputs and outputs.
- Record the suggested action, standardized parameters, policy determination, authorization, outcome, and status of reversal.
Control matrix by trust boundary
| Boundary | Representative threat | Prevent | Detect | Contain and recover |
|---|---|---|---|---|
| Interface | Prompt injection, jailbreak, session abuse | Strong identity, input isolation, file limits, rate limits | Abuse signals, anomalous sessions, cost spikes | Revoke sessions, throttle, quarantine uploads |
| Orchestration | Prompt leakage, template injection, memory poisoning | No secrets in prompts, typed templates, governed memory writes | Prompt/version traces, memory change alerts | Disable affected workflow, purge memory, restore known version |
| Context | Indirect injection, poisoned index, permission bypass | Source governance, ACL filtering, provenance, content isolation | Retrieval drift, unusual source concentration, access denials | Revoke source, rebuild index, invalidate caches |
| Model | Compromised weights, poisoned fine-tune, exhaustion | Signed artifacts, lineage, evaluation gates, quotas | Behavior regression, integrity checks, resource telemetry | Rollback model, isolate workload, rotate artifacts |
| Response & tools | Unsafe output, excessive agency, exfiltration | Schema validation, least privilege, policy checks, approvals | Tool-call audit, DLP signals, action anomaly detection | Cancel or reverse action, revoke token, notify impacted owners |
Secure reference flow
- Authenticate and classify. Establish the caller’s identity, tenant, role, intent, and risk tier.
- Retrieve with authorization. Before content enters context, apply permissions and attach source and classification metadata.
- Construct bounded context. Distinguish between trusted policies and untrusted evidence, and apply limits to tokens and sources for enforcement.
- Generate without authority. Allow the model to suggest a response or course of action, without granting itself permission.
- Validate deterministically. Analyze structured output, verify policies, validate arguments, examine data flow, and assess impact.
- Approve and execute minimally. Utilize a scoped tool identity, seek confirmation as needed, and ensure actions are either idempotent or reversible.
- Observe and learn safely. Collect security data, but ensure proper governance before storing model results in long-term memory or training datasets.
Production security checklist
- Record all inputs, data flows, trust boundaries, models, tools, and outbound connections.
- Do not include credentials, secrets, or authorization decisions in prompts or model context.
- Ensure that user and tenant permissions are enforced both prior to retrieval and prior to the execution of any tool.
- Monitor the origin and authenticity of documents, memory, embeddings, datasets, weights, and adapters.
- Validate model output with schemas, allowlists, sanitization, and parameterized interfaces.
- Utilize minimal access, temporary credentials and limit outbound network traffic for tools and agents.
- Require approval for high-impact, external, destructive, financial, or irreversible actions.
- Establish strict limits for tokens, steps, retries, duration, concurrency, and expenses.
- Red-team techniques include direct and indirect injection, retrieval of poisoned data, memory attacks, and exfiltration of tools.
- Keep records of activities, response plans, emergency shut-off mechanisms, reversal procedures, and accountability.
How to test the attack surface
Develop adversarial tests based on actual workflows instead of a standard set of jailbreak prompts. Integrate hostile instructions into various user interfaces, documents, online content, emails, retrieved data, image text, memory, and tool responses. Explore cross-tenant retrieval, unauthorized tool inputs, data leaks, misshaped output, recursive agent loops, resource-intensive requests, and substitution of dependencies.
Assess if sensitive data was accessed, if policy checks were executed, if actions received proper approval, if tool coverage was adequate yet minimal, and if the system generated valuable alerts. Reevaluate whenever there are modifications to prompts, models, retrievers, embedding models, data sources, agents, tools, permissions, or policies.
Frequently asked questions
What is the LLM security attack surface?
Security must encompass all entry points and trust boundaries within an AI application, including user interfaces, prompt orchestration, retrieved context and memory, model artifacts and inference, generated responses, and connected tools, rather than focusing solely on the model itself.
Why are prompt injections difficult to eliminate?
LLMs handle instructions and untrusted data in the same natural-language channel, making it possible for a malicious instruction to be present in various forms such as user input, documents, webpages, retrieved memory, or tool output. To defend against this, isolation, authorization, constrained tools, validation, and monitoring are necessary, rather than solely relying on a prompt filter.
What is the most important control for an AI agent?
Limit access privileges to only what is necessary outside the defined boundaries. Assign specific credentials to each tool, verify all inputs, limit resource access, establish budgets and rate limits, and mandate approval for significant actions.
How should organizations test LLM application security?
Evaluate every trust boundary and end-to-end workflow by utilizing adversarial prompts, poisoned documents, unauthorized retrieval attempts, malformed model outputs, tool abuse scenarios, data exfiltration attempts, resource exhaustion, and compromised dependency simulations. Conduct retesting following any modifications to the model, prompts, tools, data, or policies.
Authoritative guidance
This guide’s risk coverage aligns with the OWASP Top 10 for LLMs and Generative AI Applications (2025)The list includes prompt injection, sensitive information disclosure, supply-chain risk, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Governance and lifecycle recommendations are influenced by these factors as well. NIST AI Risk Management Framework: Generative AI Profile.