Dataknobs
Dataknobs
Generative AI Security Explained

LLM Security: Attack Paths, Defenses and Launch Readiness

Security for large language models is not just about the model itself, but rather a comprehensive system issue. This visual guide tracks the entire process from user input and prompt organization to context retrieval, model operation, output generation, and tool integration—transforming potential risks into actionable safeguards for RAG systems and AI helpers.
Attack Surface
Prompt Injection
RAG Exfiltration
Excessive Agency
Launch Checklist
Visual security agenda

Six questions every LLM application must answer

The ten visuals provided create a comprehensive security curriculum that covers topics such as understanding the differences between LLM applications and traditional software, mapping attack surfaces, studying injection and data exfiltration techniques, managing excessive permissions, and testing defenses prior to deployment.

LLM security agenda listing LLM versus software security, attack surface, injection, data location, excessive agency, and security defenses
The discussion shifts from comparing LLM and software security architecture to exploring protective measures for AI systems when exposed to hostile text.
Security in one paragraph

To mitigate the risk of an LLM following malicious or misleading content, enforce user-level access controls before accessing, avoid sharing secrets in prompts, limit tools and outbound destinations, validate all actions, require human confirmation for critical operations, and maintain searchable logs for detection and response.

Detailed guide

A complete LLM security walkthrough

Every section combines a visual agenda with the corresponding architecture, risk, and control decisions.

1. LLM vs software security

How language models blend the traditional distinction between executable instructions and data.

2. Attack surface

The flow of risk passes through interfaces, orchestration, context, models, responses, and tools.

3. Injection

The impact of jailbreaks and indirect instructions in retrieved content on behavior.

4. Data location

Sensitive data may escape the trust boundary and be retained in logs when entering prompts.

5. Excessive agency

The blast radius increases when tools, credentials, destinations, and autonomous loops are overpowered.

6. Security defenses

Permission controls, isolation, validation, human approval, monitoring, and rehearsed response.

Attack surface

Map the complete LLM application, not only the model

An LLM application consists of trust boundaries where risk originates at the interface, traverses orchestration and memory, accesses retrieved context, interacts with the model, and may manifest as real-world consequences through generated output or tools.

  • Interface: direct prompt injection, jailbreaking, account abuse, and session abuse.
  • Orchestration: system-prompt leakage, prompt-template injection, and poisoned memory.
  • Context: Documents are indirectly injected, index entries are poisoned, and permission is bypassed during retrieval.
  • Model: compromised weights, poisoned fine-tuning data, and unbounded resource use.
  • Response and tools: improper output handling, excessive agency, and exfiltration through connected capabilities.
LLM attack surface across interface, orchestration, context, model, response, and tools
Security may be compromised if an AI system obediently processes malicious text within an operational pipeline.
Trust boundaries

Turn every vertical boundary into an enforced control

The attack-surface visual reappears due to the intricate architecture, necessitating a thorough review at every transition point to validate identity, provenance, authorization, data classification, and permitted actions.

  • Authorize access before retrieving the document instead of querying the model for user permissions.
  • Categorize system policies, developer guidelines, user feedback, evidence sources, memory storage, and tool reactions based on their origin and level of trust.
  • Validate output before rendering, executing, storing, or forwarding it.
  • Assign specific credentials, schemas, destination rules, and budgets to each individual tool.
Key question: What new data or capabilities can be obtained if the model is altered at this boundary?
Repeated LLM attack-surface diagram emphasizing trust boundaries between system components
Deliberately repeated: the safety of architecture is enhanced when trust boundaries are clearly defined and upheld.
Architectural challenge

LLMs collapse the code-and-data boundary

Traditional application security typically relies on the separation of executable code from data. An LLM interprets system rules, user requests, retrieved documents, tool results, and memory as language, making prompt injection distinct from SQL injection due to the absence of a universal equivalent of a prepared statement in natural language that ensures data integrity.

The model can read any content, such as documents, webpages, emails, tool results, code comments, or image text, which may contain instructions. The main goal is to restrict what the model can access, reveal, execute, alter, or transmit.

Five challenges in LLM security including prompt structure, injection, perimeter expansion, architectural weakness, and containment
Prompt injection originates from the natural processing of language, meaning that relying solely on one filter or prompt patch is not enough for complete defense.
Injection types

Direct attacks are visible; indirect attacks hide in data

Direct injection When a user tries to bypass rules, extract prompts, or redirect an assistant, they are said to be "jailbreaking" the system, within the limitations of their existing access.

Indirect injection The model is embedded in different types of materials for the model to interpret later, such as documents, webpages, emails, code comments, tool responses, index entries, or visually hidden PDF text. A genuine user inadvertently activates the payload while performing a routine task, allowing the payload to operate with the assistant's permissions instead of the attacker's.

Comparison of direct prompt injection and indirect injection hidden in RAG content
RAG increases the attack surface by incorporating external content into the model's instruction-following context.
RAG exfiltration

Follow the plant–wait–trigger–obey–exfiltrate chain

  1. Plant: Harmful commands are inserted into a collaborative file, ticket, website, or any other accessible source.
  2. Wait: the content sits in the index beside legitimate material.
  3. Trigger: A regular inquiry from a user leads to the selection of a contaminated passage during retrieval.
  4. Obey: The model views the hidden text as guidance and operates with the consent of the user or assistant.
  5. Exfiltrate: Data exits via a variety of channels, including an externally loaded resource, user-clicked link, or outbound tool call.

Break the chain by controlling access points: manage sources, grant permission for retrieval, limit context, restrict outbound destinations, verify output, and mandate approval for sensitive actions.

Five-step RAG exfiltration chain: plant, wait, trigger, obey, and exfiltrate
Preventing model-supplied outbound URLs and resources can eliminate a common exfiltration pathway, even in the event of an injection impacting model performance.
Four leak locations

Know where confidential data can escape

  • Answer leakage: The assistant retrieves or summarizes data that the user is unauthorized to access.
  • System-prompt leakage: Users access internal instructions, business logic, integrations, routing rules, and embedded secrets.
  • Provider boundary: Prompts and collected data exit the organization's system for model analysis.
  • Logs and telemetry: Prompts, responses, retrieved passages, tool arguments, and errors can generate extra duplicates of sensitive information.

Reduce sensitive information exposure, avoid including credentials or secrets in prompts, assess provider retention and training options, implement log redaction, and establish suitable access and retention regulations.

Four ways LLM application data can leak through answers, system prompts, providers, and logs
The ultimate outcome is a single instance of the data; prompts, provider processing, and logs are stored in distinct security and governance areas.
Defense in depth

Build five layers that reduce reach and consequence

1
Reduce reach

Implement user-specific permission controls and utilize the principle of least privilege when managing access to data, tools, destinations, and credentials.

2
Break exfiltration

Avoid loading random images, URLs, scripts, or other external resources provided by the model.

3
Human responsibility

Confirmation is necessary for important actions like payments, deletions, permission changes, and disclosing information externally.

4
Filter and constrain

Utilize injection detection, typed schemas, output validation, moderation, sandboxing, quotas, and allowlists for security measures

5
Watch and rehearse

Retrieve logs and utilize tool chains, detect abnormal behavior, simulate red-team scenarios, and train incident response procedures.

Five layers of LLM defense: reduce reach, break exfiltration, human responsibility, constrain, and monitor
The most powerful controls are positioned beyond the model's reach and continue to be effective even when the model is given a specific command.
Launch readiness

Checklist for an LLM or AI assistant launch

Data and access

  • Filter retrieval by the requesting user’s permissions.
  • Apply consistent access, lifecycle, and deletion rules to vector indexes as with source documents.
  • Keep secrets and confidential data out of system prompts.
  • Find out where prompts are stored in the document and how the model provider stores or utilizes submitted content.

Actions and output

  • Give each tool a separate least-privilege credential.
  • Require confirmation before payments, deletion, external communication, or permission changes.
  • Validate model output before rendering, execution, storage, or forwarding.
  • Do not automatically load model-created external images or URLs.

Operations

  • Logging prompts, retrieving RAG, generating model output, making policy decisions, granting approvals, and calling tools.
  • Make security logs searchable and access-controlled.
  • Limit tokens, steps, retries, elapsed time, spend, and external calls.
  • Enhance incident response to include assistants, indexes, memory, models, prompts, and tool credentials.
Checklist for launching an LLM or AI assistant across data access, actions and output, and operations
A launch review should ensure access, actions, output handling, observability, budgets, and incident response are verified, not just model quality.
Takeaways

Four principles to carry into every design review

1
Instructions and data share a channel

The model does not offer a guaranteed security boundary separating trusted commands from untrusted language.

2
Contain rather than promise a cure

Expect that certain injections will be successful and limit the data, actions, destinations, and independence they can access.

3
Enforce permission at retrieval

The model must only display content that the authenticated user has permission to access.

4
Keep humans in critical decisions

Informed confirmation should be required for high-impact actions after showing the exact action and consequences.

Four LLM security takeaways: instruction and data, containment, permission at retrieval, and human approval
Permission control and containment are the most reliable defenses when data has the power to impact behavior, much like code.

Frequently asked questions

What is the LLM application attack surface?

The attack surface encompasses user interfaces, prompt orchestration and memory, retrieved context, model weights and inference behavior, generated responses, as well as all connected tools or downstream systems.

Why is RAG vulnerable to indirect prompt injection?

RAG brings in outside content to the model's context, where a tainted document or webpage may include commands that the model wrongly identifies as safe, even if a genuine user made the request.

How can LLM applications prevent data exfiltration?

Enforce user-level permissions prior to accessing data, prevent unauthorized outbound URLs and model-supplied resources, restrict tools and destinations, verify output, and mandate confirmation for sensitive or external operations.

What should be checked before launching an AI assistant?

Confirming access to user-specific data, controlled indexes, prompt without secrets, tools with minimal privileges, validation prior to rendering or execution, human verification for important actions, easily searchable logs, set resource limits, alerts, and protocols for responding to incidents.

Authoritative guidance

The control themes in this visual guide align with the OWASP Top 10 for LLMs and Generative AI ApplicationsGovernance and lifecycle practices are influenced by various factors such as prompt injection, sensitive information disclosure, data and model poisoning, improper output handling, excessive agency, system-prompt leakage, vector and embedding weaknesses, and unbounded consumption. NIST AI Risk Management Framework: Generative AI Profile.