AI-agent security is broader than model safety. A production agent may retrieve enterprise data, call APIs, modify records, send messages, execute code, retain memory, hand work to other agents, and operate over long-running workflows. Security therefore has to cover the complete chain from input and context through authorization, execution, persistence, and audit.
The security question is not only whether the model can be manipulated. It is what the system permits after the model produces a decision.
1. Security model: protect the system, not only the model
Model-level safeguards are useful, but they are not an authorization boundary. A secure agent architecture assumes that a model can misunderstand intent, over-trust retrieved content, select an inappropriate tool, or produce an invalid action request. Security controls should remain effective even when that happens.
A useful control chain is:
Input / event
↓
Identity + source classification
↓
Agent reasoning
↓
Proposed action
↓
Deterministic policy + authorization
↓
Approval when required
↓
Constrained execution
↓
Telemetry + audit + response
2. Treat untrusted content as data, not authority
Prompt injection is a content-originated instruction-manipulation problem. The relevant content may come from a user, webpage, document, email, retrieved passage, tool output, or peer agent. Because those inputs eventually become model context, the system must preserve the distinction between approved authority and untrusted evidence.
Direct and indirect prompt injection should therefore be handled as a trust-boundary problem, not only as a text-classification problem.
- System and policy instructions define approved authority.
- User instructions are evaluated within the user’s scope.
- Retrieved content and tool output are evidence and must not silently acquire control authority.
- Peer-agent messages require authenticated provenance and scoped delegation.
Content scanning can reduce risk, but policy enforcement must still occur before tools execute.
3. Design around explicit trust boundaries
For security architecture reviews, it is useful to define six boundaries that can be assessed independently:
| Boundary | Security question | Representative controls |
|---|---|---|
| Content | Which inputs are trusted, untrusted, or sensitive? | Source classification, provenance, parsing controls, retrieval filters |
| Identity | Who is acting and on whose behalf? | User identity, agent identity, application identity, workload identity |
| Capability | Which tools, data, and actions may this agent use? | Allowlists, scopes, role constraints, tool schemas |
| Execution | Which proposed actions are actually permitted? | Policy engine, transaction limits, approvals, sandboxes, network controls |
| Persistence | What may be written to memory or durable state? | Memory classes, provenance, expiry, validation, correction and deletion |
| Delegation | What authority may pass to another agent or service? | Authenticated handoffs, scoped task context, correlation IDs, delegation policy |
Least agency
Choose the lowest-autonomy pattern that satisfies the business requirement:
- AI recommends; a person executes.
- AI prepares; a person approves; the system executes.
- AI executes only bounded, reversible actions.
- Higher-impact actions require stronger authorization and review.
Intelligence and authority should be controlled independently. A stronger model should not automatically receive broader permissions.
4. Separate user identity, agent identity, and execution authority
A user and the user’s agent should not be treated as the same security principal. A secure decision may need to evaluate:
user identity
AND agent identity
AND application identity
AND tool permission
AND resource scope
AND task scope
AND approval stateGive agents distinct identities, owners, environments, approved tools, data scopes, action limits, and credential lifetimes. Prefer short-lived delegated credentials over long-lived secrets, and avoid placing powerful credentials in model context.
User authority should normally be an upper bound, not a direct grant. A user may be able to perform a privileged operation manually while their autonomous agent is limited to recommending or preparing the same operation.
5. Treat tools and MCP integrations as application security boundaries
Tool access changes the risk profile because a model decision can become a database write, API call, message, file operation, deployment, or transaction. The right control is not simply “the model should not call this tool.” The execution layer must determine whether the specific call is allowed.
For each tool or MCP server, inventory and control:
- server and publisher identity, version, and ownership;
- approved tools and schemas;
- requested scopes and credentials;
- allowed network destinations;
- argument validation and resource-level authorization;
- logging, update policy, and review status.
Tool descriptions and tool output should be treated as untrusted application data. Sensitive operations should be authorized independently of the language model.
6. Govern memory and multi-agent delegation
Memory is persistent state and should be secured like other durable application data. A memory write should carry source, confidence, sensitivity, ownership, expiry, and business relevance. The system should support correction, deletion, invalidation, and retention policy.
Separate memory classes because they have different trust and retention requirements:
- working/session memory;
- user preferences;
- enterprise knowledge;
- episodic history;
- learned heuristics or agent-specific state.
For multi-agent systems, authenticate agents to each other and preserve sender identity, task identity, authorization, message integrity, replay protection, and correlation IDs. Forward only the context required for the delegated task rather than complete histories.
7. Separate reasoning, control, and execution planes
A strong reference architecture treats the model as one component inside a governed system:
USER / EVENT
↓
IDENTITY + CONTENT GATE
↓
AGENT HARNESS
(model / retrieval / memory / planner)
↓
PROPOSED ACTION
↓
POLICY + AUTHORIZATION ENGINE
↓
AUTO-APPROVED or HUMAN REVIEW
↓
EXECUTION GATEWAY / SANDBOX
↓
ENTERPRISE SYSTEM
↓
TELEMETRY + AUDIT + RESPONSEThe model can be probabilistic; authorization does not need to be. Policy should be implemented as executable rules rather than natural-language guidance alone.
Risk-based approval
| Action class | Default posture | Examples |
|---|---|---|
| Read-only / low consequence | Autonomous with logging | Search, retrieve, summarize, classify |
| Reversible write | Bounded execution | Draft update, create proposed record, soft-delete request |
| Material business action | Policy check + threshold-based approval | Refund, pricing change, external communication |
| Privileged or irreversible | Explicit approval and stronger identity verification | Payment, production deployment, account closure, destructive change |
Reversibility should be treated as a design property. Prefer draft-before-send, staged deployment, soft delete, and proposed change before commit where possible.
8. Make runtime security controls explicit
Important runtime controls include:
- tool allowlists and validated schemas;
- resource- and action-level authorization;
- transaction and spend limits;
- step, token, delegation, and external-request budgets;
- network egress allowlists and private endpoints;
- retrieval authorization and source eligibility;
- memory write policy and retention;
- human approval for defined consequence classes;
- execution sandboxes for generated code or high-risk tools.
Budgets are security controls as well as cost controls: they limit loops, runaway workflows, excessive delegation, and uncontrolled external activity.
9. Observability is part of the security boundary
Capture enough evidence to reconstruct what happened without relying on hidden model reasoning. Useful telemetry includes:
Identity
Task ID, user identity, agent identity, application identity, environment.
Context
Approved goal, source references, retrieved evidence, memory reads, peer-agent handoffs.
Decision
Model/provider version, proposed tool, arguments, policy result, approval decision.
Execution
Credential used, resource affected, execution result, external destination, rollback state.
Persistence
Memory writes, durable state changes, downstream notifications, delegation chain.
Operations
Latency, cost, anomalies, policy violations, security events, business outcome.
Behavioral baselines can surface abnormal tool usage, unusual destinations, unexpected workflow sequences, or atypical transaction values.
10. Security evaluation must be continuous
Security testing should include direct and indirect prompt injection, malicious or malformed documents, untrusted tool output, unauthorized tool selection, excessive privilege, invalid memory writes, deceptive peer-agent messages, loops, policy-engine failure, network failure, and tool failure.
The acceptance criterion should not be only whether the model refuses. The architecture should prevent a forbidden action even when the model proposes it.
Run regression testing after changes to:
- model or provider version;
- system prompt or planner;
- tool or MCP-server inventory;
- retrieval and memory policies;
- authorization and approval rules;
- agent topology or delegation model.
11. Design incident response into the platform
Security teams should be able to rapidly constrain the system without rebuilding it. Useful controls include:
- disable an agent or workflow;
- disable a tool or MCP integration;
- revoke delegated credentials;
- reduce an agent to read-only mode;
- disable external network access;
- force human approval;
- invalidate or quarantine memory;
- preserve execution evidence;
- roll back reversible actions;
- identify downstream agents and systems affected.
A practical response sequence is:
detect → constrain → preserve evidence → revoke → rollback → investigate → remediate → re-enable12. Practical implementation sequence
- Inventory agents, tools, MCP servers, data access, credentials, and external destinations.
- Classify actions by consequence and reversibility.
- Reduce agency where a recommendation or approval workflow is sufficient.
- Establish distinct agent identities and short-lived scoped credentials.
- Put deterministic policy and authorization between reasoning and execution.
- Sandbox code execution and constrain network access.
- Govern retrieval, memory, and peer-agent handoffs.
- Add traceable execution evidence and anomaly monitoring.
- Run adversarial and regression testing before and after material changes.
- Maintain kill switches, rollback procedures, and credential revocation paths.
13. Where KONTROLS and KNOBS fit
KONTROLS can operationalize identity, data access, tool permissions, execution limits, approvals, network restrictions, memory policy, and audit evidence.
KNOBS can expose autonomy level, approval threshold, transaction limit, tool allowlist, retrieval scope, memory retention, step budget, delegation depth, network access, and confidence thresholds as versioned operating parameters.
The objective is not to make the model responsible for security policy. It is to make security policy explicit, independently enforced, measurable, and adjustable as the workload changes.
Frequently asked questions
How is AI-agent security different from model security?
Model security focuses on model behavior and inputs. AI-agent security must additionally control identity, credentials, tools, retrieval, memory, delegation, authorization, execution and persistent side effects.
Should the model decide whether an action is authorized?
No. The model can recommend an action, but authorization should be performed by deterministic policy and identity controls outside the model, especially for consequential actions.
How should retrieved documents and tool output be treated?
As untrusted evidence unless a separate trust mechanism says otherwise. Their presence in model context should not grant them the authority of system policy or an authenticated user instruction.
How should MCP integrations be secured?
Treat them as application integrations: authenticate endpoints, inventory approved tools, minimize scopes, validate metadata and arguments, restrict network access, log execution, and independently authorize sensitive actions.
What should be logged for security investigations?
At minimum: task, user and agent identities; approved goal; model version; sources; tool calls and arguments; authorization and approval results; execution results; memory writes; delegations; external destinations; policy events; and resulting business impact.
KONTROLS + KNOBS
Make agent authority, runtime policy, approvals, tool scope, memory policy and operating limits explicit, enforceable and measurable.