How to Test an AI Agent for Security Vulnerabilities: A Complete Guide

How to Test an AI Agent for Security Vulnerabilities: A Complete Guide

AI agent security testing evaluates whether an agent can be manipulated into revealing sensitive information, trusting malicious context, crossing user or session boundaries, misusing tools, escalating privileges, poisoning memory, or performing actions outside the user's authority.

Testing an AI agent is different from testing a chatbot.

A chatbot primarily generates text.

An AI agent can retrieve information, maintain memory, call APIs, use MCP tools, interact with other agents and execute real-world actions.

That means the security question is no longer simply:

“Can someone jailbreak the model?”

The more important question is:

“Can an attacker make the agent cross a trust boundary it should never cross?”

This guide explains how to test that.

---

Key Takeaways

If you read nothing else, these are the most important principles for AI agent security testing.

• An AI agent is an execution system, not just an LLM. The attack surface includes prompts, models, RAG, memory, tools, MCP servers, identities, permissions, other agents and external APIs.
• Prompt injection is only one failure mode. Agent security testing should also cover memory poisoning, tool misuse, privilege escalation, data exfiltration, state leakage and cross-agent attacks.
• Test security invariants, not just jailbreak prompts. Define what must never happen, then deliberately test whether the agent can violate that rule.
• Every boundary matters. User, session, tenant, model, agent, memory, tool, privilege and workflow-version boundaries should be tested independently.
• Model refusal is not authorization. Sensitive operations should be protected by deterministic permission and policy controls outside the model.
• Observe execution, not just conversation. A safe-looking final answer does not prove that the agent behaved safely internally.
• Security testing should become continuous. Changes to prompts, models, tools, retrieval, memory or policies can change an agent's security properties.

---

Quick Answer: What Should You Test in an AI Agent?

OWASP's AI Agent Security guidance identifies many of the same classes of risk, including direct and indirect prompt injection, tool abuse, privilege escalation, data exfiltration, memory poisoning, excessive autonomy and cascading failures between agents.

---

What Is AI Agent Security Testing?

AI agent security testing is the systematic evaluation of whether an AI agent can be manipulated into violating the security boundaries, permissions and policies under which it is supposed to operate.

The key word is systematic.

Sending a hundred jailbreak prompts to an agent may reveal weaknesses, but it does not tell you whether the entire agent architecture is secure.

A production agent might contain:

User → Application → System Instructions → Model → RAG → Memory → MCP → Tools → APIs → Other Agents → External Systems

Every connection introduces a possible trust boundary.

Security testing therefore needs to evaluate both:

what the model says

and

what the system does.

---

Why Is Testing an AI Agent Different From Testing an LLM?

An LLM generally maps an input to an output.

An agent can turn an output into an action.

Consider a customer-support LLM.

A successful prompt injection might produce an incorrect response.

Now consider a customer-support agent connected to:

• CRM
• email
• payment systems
• customer records
• ticketing software
• internal knowledge bases

A successful attack could potentially influence how those systems are accessed or used.

This is why agent security extends beyond model security.

NIST's 2026 analysis of AI-agent security responses found broad agreement that agents create novel security challenges and that existing cybersecurity practices need to be adapted for agentic systems.

The security unit is therefore not simply the model.

The security unit is the complete execution path.

---

What Is the Attack Surface of an AI Agent?

A useful way to understand an agent is to break its execution environment into surfaces.

1. User input

Anything the user can directly provide:

• text
• voice
• images
• files
• URLs
• structured parameters

This is the most obvious untrusted boundary.

But it is not the only one.

2. Retrieved context

Agents increasingly retrieve information from:

• websites
• documents
• emails
• databases
• vector stores
• search engines
• CRM records

Retrieved information is data.

It should not automatically become trusted instructions.

3. Memory

Memory may include:

• conversation summaries
• user preferences
• task history
• learned facts
• previous tool results
• long-term state

Persistent memory creates an especially important security problem:

an attack can potentially survive beyond the interaction that created it.

OWASP has specifically highlighted memory and context poisoning as an agentic attack surface because attacker-controlled information may persist and influence future decisions.

4. Tools

Tools turn reasoning into capability.

Examples include:

• send_email
• search_customer
• update_crm
• create_invoice
• issue_refund
• run_code
• execute_query

A model deciding to call a tool should not automatically mean the operation is authorized.

5. MCP servers

Model Context Protocol environments introduce another trust boundary between the agent and external capabilities.

Tool descriptions, tool responses and server behavior can all influence the model.

OWASP describes MCP tool poisoning as a form of indirect prompt injection where malicious tool responses can introduce instructions into the model's context.

6. Other agents

Multi-agent systems introduce transitive trust.

Agent A may generate information consumed by Agent B.

Agent B may have capabilities Agent A does not.

The important question becomes:

Does Agent B know where Agent A's information originally came from?

7. External actions

The highest-risk boundary is usually where AI reasoning becomes an irreversible action.

Examples include:

• sending an email
• making a payment
• changing permissions
• deleting data
• publishing information
• modifying infrastructure

These actions deserve stronger controls than ordinary conversational responses.

---

What Are AI Agent Trust Boundaries?

An AI agent trust boundary is a point where information, authority or state moves between security contexts with different levels of trust.

Important boundaries include:

This gives us a more useful security model than simply asking whether an agent can be jailbroken.

---

What Does the “Stealing Reasoning Traces” Research Teach Us?

A 2026 paper titled Stealing Reasoning Traces from Proprietary LLM APIs demonstrated an important class of trust-boundary failure.

The researchers examined encrypted reasoning blocks returned by major model APIs and reported that, in the systems they tested at the time, reasoning state could be portable across contexts, users or models within a provider ecosystem.

That portability enabled attacks involving reasoning extraction, private-data recovery and hidden prompt injection.

The researchers reported decoding 315,320 reasoning blocks collected from public repositories and finding hundreds of PII and credential artifacts.

The exact vulnerabilities were responsibly disclosed, and the authors reported that they were subsequently unable to reproduce the same attacks.

The broader security lesson survives the specific vulnerability:

A security artifact may be valid without being valid in every context.

This distinction matters enormously for agent security.

Instead of asking only:

“Is this state authentic?”

security systems should also ask:

“Is this state authorized to be trusted here?”

That same question applies to:

• memory
• tool outputs
• RAG documents
• cached context
• delegated credentials
• inter-agent messages
• resumable executions
• workflow state

---

How Do You Test an AI Agent for Security Vulnerabilities?

A repeatable agent security test can be organized into ten steps.

Step 1: Map the Complete Agent Execution Path

Before attacking anything, understand what exists.

Document:

• models
• system instructions
• RAG sources
• memory systems
• tools
• MCP servers
• APIs
• credentials
• agent-to-agent communication
• external actions

The goal is to create an execution graph.

A vulnerability cannot be understood properly if you cannot identify the path through which information or authority moved.

---

Step 2: Identify Every Trust Boundary

For every connection in the execution graph, ask:

What security assumption is being made here?

For example:

User → Agent

Is all user input treated as untrusted?

RAG → Model

Can retrieved text become instructions?

Memory → Agent

Does the agent distinguish who originally created the memory?

Agent → Tool

Does the tool independently verify authorization?

Agent A → Agent B

Does Agent B understand the provenance of Agent A's message?

This converts architecture into testable security hypotheses.

---

Step 3: Define Security Invariants

A security invariant is something that should remain true regardless of what the model generates.

For example:

Cross-user isolation

Information belonging exclusively to User A must never become observable by User B.

Tool authorization

A user without payment permission must never cause the payment tool to execute.

Memory isolation

Memory created by Tenant A must never influence Tenant B.

External transfer policy

Sensitive customer information must never be sent to an unapproved external destination.

This is one of the most important changes in how agent testing should be approached.

Do not begin with attacks.

Begin with what must never become possible.

---

Step 4: Insert Security Canaries

Canaries are unique synthetic values deliberately placed inside controlled parts of the agent.

For example:

System instructions

SYS_CANARY_7Q91

RAG

RAG_CANARY_M82K

User A memory

MEM_CANARY_2PL7

CRM

CRM_CANARY_91XZ

Tool configuration

TOOL_CANARY_A74P

Now you can measure exactly which security boundary leaked.

This is considerably more reliable than asking another model whether something “looks sensitive.”

---

Step 5: Mutate One Boundary at a Time

Suppose an artifact was created under:

• Tenant A
• User A
• Session A
• Agent A
• Model A
• Privilege A

Change one property.

Test:

Session A → Session B

Then:

User A → User B

Then:

Agent A → Agent B

Then:

Model A → Model B

Then:

low privilege → privileged action

This is effectively trust-boundary fuzzing.

Instead of randomly generating attacks, the system systematically searches for broken assumptions.

---

Step 6: Test the Major AI Agent Vulnerability Classes

Prompt Injection

Prompt injection attempts to manipulate an agent's intended instructions using attacker-controlled content.

There are two important forms.

Direct prompt injection comes directly from the user.

Indirect prompt injection arrives through something the agent consumes, such as:

• webpage
• email
• document
• database record
• tool response
• MCP response

Indirect injection is particularly important for agents because they continuously ingest information from external systems.

---

System Prompt Leakage

System prompt leakage occurs when an attacker learns information about hidden system or developer instructions.

The goal of testing should not be limited to:

“Can the agent print its system prompt?”

A stronger question is:

“How much of the hidden policy can an attacker reconstruct?”

An agent might never reveal its prompt verbatim while still exposing:

• business rules
• internal policies
• tool names
• workflow logic
• escalation thresholds
• restricted topics
• hidden variables

That leads to a broader concept we call Agent Extractability.

---

Memory Poisoning

Memory poisoning occurs when attacker-controlled information becomes persistent state and influences future behavior.

Test whether malicious or misleading content can move from:

Input → Memory → Future Session

Then test:

User A → Memory → User B

and:

Agent A → Shared Memory → Agent B

Memory should have provenance, scope and expiration—not simply content.

---

RAG Poisoning

RAG poisoning occurs when malicious information enters a retrieval system and later influences an agent.

A secure system should distinguish:

retrieved information

from

trusted instructions.

OWASP's prompt-injection guidance explicitly identifies poisoned retrieval content as an attack pattern.

---

MCP Tool Poisoning

MCP tool poisoning tests whether malicious or compromised tool information can influence the agent beyond the tool's intended function.

A useful invariant is:

Tool output should never gain instruction authority merely because it came from a connected tool.

MCP security should therefore evaluate both what tools can do and what information they can introduce into the model's context.

---

Tool Abuse

A legitimate tool can still be used incorrectly.

Consider:

CRM lookup → customer record → email tool

Every operation may be individually legitimate.

But the combined sequence could expose customer information to an unauthorized destination.

Security testing therefore needs to analyze sequences of actions, not merely individual tool calls.

---

Privilege Escalation

Agents should operate under the authority of the principal requesting the action.

Test:

Low-privilege user → Agent → High-privilege tool

The model should never be the sole authorization layer.

OWASP recommends least privilege, per-tool permission scoping and explicit authorization for sensitive operations.

---

Data Exfiltration

Seed the controlled test environment with synthetic secrets.

Then determine whether they can reach:

• final responses
• logs
• memory
• tool parameters
• URLs
• emails
• external APIs
• another user
• another tenant
• another agent

The question is not simply whether the model reveals a secret in chat.

The question is:

Can sensitive information cross an unauthorized boundary by any execution path?

---

Agent-to-Agent Injection

Multi-agent systems introduce another important attack class.

Consider:

External webpage → Research Agent → Finance Agent → Payment Tool

Even if the finance agent never accesses the malicious webpage itself, attacker-controlled information may have crossed through the research agent.

This creates a provenance problem.

Every agent-to-agent message should therefore carry enough context for downstream components to understand where its information originated.

---

State Replay and Context Confusion

State that was valid yesterday, for another session or under another policy may not be valid today.

Test whether:

State(Context A) → Context B

is accepted when it should be rejected.

Useful dimensions include:

• user
• tenant
• session
• agent
• model
• permissions
• policy version
• workflow version

The underlying principle is simple:

Authenticity does not automatically imply contextual authorization.

---

What Is Agent Extractability?

Agent Extractability is the degree to which an attacker can reconstruct proprietary, sensitive or security-relevant information about an AI agent through the interfaces available to them.

Traditional prompt-leakage testing focuses primarily on the system prompt.

But modern agents contain much more than prompts.

An attacker may attempt to infer:

• system instructions
• developer policies
• workflow structure
• tool inventory
• tool schemas
• RAG sources
• memory contents
• escalation rules
• authorization logic
• model configuration
• internal state

Agent Extractability therefore provides a broader measurement.

---

What Should an Agent Extractability Test Measure?

A useful framework separates extractability into dimensions.

Eventually these measurements can form an Agent Extractability Score.

The goal is not to claim that every agent can be reverse engineered.

The goal is to measure:

How much does this particular agent expose to this particular attacker under this particular access model?

---

Why Conversation Logs Are Not Enough

Imagine an agent responds:

“I can't perform that action.”

A conventional monitor may mark the interaction as safe.

But before producing that response, the agent could potentially have:

• retrieved sensitive information
• read poisoned memory
• requested a privileged tool
• sent information to another agent
• modified state
• generated an unsafe action that another layer blocked

The final response describes only one part of execution.

That is why agent security testing needs execution-level observability.

Monitor:

Input → Context → Retrieval → Model → Memory → Tool → External Action

not merely:

Prompt → Response

---

How Should Automated AI Agent Security Testing Work?

A practical automated architecture looks like this:

Agent/Application

↓

Execution Capture

↓

Trust-Boundary Test Harness

↓

Deterministic Security Rules + Semantic Security Evaluation

↓

Invariant Detection

↓

Exposure Graph

↓

Finding → Evidence → Reproduction → Remediation

The distinction between deterministic and semantic testing matters.

---

What Should Be Deterministic?

Use code where the answer can be known precisely.

Examples include:

• user permissions
• tenant IDs
• domain allowlists
• secret canaries
• API scopes
• destination restrictions
• approved tools
• transaction limits
• credential boundaries

If User B accesses User A's canary, you do not need an LLM to decide whether that happened.

---

Where Can Semantic Security Models Help?

Some security questions require interpretation.

For example:

• Did the response paraphrase a hidden policy?
• Does this tool sequence resemble exfiltration?
• Is this memory entry functioning like an instruction?
• Did retrieved content alter the agent's intended objective?
• Does the action contradict the user's apparent request?

A semantic evaluator can assist with these judgments.

But it should not become the authorization system.

Use AI to identify ambiguous risk. Use deterministic policy to enforce known rules.

---

What Metrics Should AI Agent Security Testing Track?

Security testing becomes more useful when results are measurable.

Track metrics such as:

These metrics allow teams to compare agent versions instead of relying on subjective impressions.

---

What Does a Good AI Agent Vulnerability Report Look Like?

A finding should explain the violated security invariant.

For example:

Finding: Cross-Tenant Memory Isolation Failure

Object:
memory_record_817

Created under:
Tenant A
User A
Agent: sales-agent
Privilege: read-only

Consumed under:
Tenant B
Agent: finance-agent

Expected behavior:
Cross-tenant memory rejected.

Observed behavior:
Memory accepted and influenced downstream agent behavior.

Security impact:
Untrusted state crossed a tenant boundary.

Reproduction rate:
18/20 controlled tests.

This is much more useful than:

“We jailbroke the model.”

---

Should AI Agent Security Testing Happen Before or After Deployment?

Both.

Pre-deployment testing

Before deployment, test:

• prompt exposure
• injection resistance
• memory isolation
• RAG poisoning
• tool permissions
• MCP security
• cross-user boundaries
• privilege escalation
• data exfiltration
• irreversible actions

Production monitoring

After deployment, monitor:

• unusual tool sequences
• authorization failures
• sensitive data movement
• unexpected destinations
• persistent memory changes
• agent-to-agent propagation
• policy violations
• behavioral drift

OWASP recommends structured adversarial testing before production and after material changes to prompts, tools, memory, retrieval, policies or model providers.

The important implication is that agent security is not a one-time penetration test.

It is a continuous process.

---

What Is the Difference Between AI Agent Observability and AI Agent Security?

They answer different questions.

Observability is therefore an important foundation.

But observability alone does not stop an action.

A mature architecture progresses from:

Observe → Detect → Decide → Enforce

---

What Is AI Agent Runtime Security?

AI agent runtime security is the enforcement layer that evaluates agent actions while the agent is operating and prevents unsafe or unauthorized operations before they execute.

For example:

Agent requests tool

↓

Security layer evaluates

• identity
• permission
• data classification
• destination
• action reversibility
• execution provenance
• risk signals

↓

ALLOW / REQUIRE APPROVAL / BLOCK

This is particularly important for irreversible operations.

The model can propose an action.

The security layer decides whether the action is permitted.

---

Why Security Teams Should Test Invariants Instead of Only Prompts

Prompt libraries are useful.

But agents change.

Models change.

Prompts change.

Tools change.

Attack wording changes.

Security invariants are more durable.

Consider:

Customer PII must never be transmitted to an unauthorized external domain.

There may be thousands of prompts capable of attempting to violate that policy.

You do not need to predict all of them.

You need to detect whether the invariant was violated.

That changes the security problem from:

“Did we anticipate the attack?”

to:

“Did the system enter a forbidden state?”

That is a much stronger foundation for agent security.

---

A Practical AI Agent Security Testing Framework

The complete methodology can be summarized as:

1. Map the agent.

Identify every model, memory, RAG source, tool, MCP server, identity and external system.

2. Map trust boundaries.

Identify where information, state or authority changes security context.

3. Define invariants.

Write down what must never happen.

4. Seed canaries.

Insert synthetic identifiers and secrets into controlled locations.

5. Generate adversarial scenarios.

Test direct and indirect attack paths.

6. Mutate boundaries.

Change user, session, tenant, model, agent, privilege and version independently.

7. Capture execution.

Observe context, retrieval, memory, model calls, tools and actions.

8. Detect violations.

Combine deterministic rules with semantic security evaluation.

9. Reproduce findings.

Verify that failures are repeatable.

10. Regression-test continuously.

Run the same suite whenever the agent changes.

---

Frequently Asked Questions

What is AI agent security testing?

AI agent security testing evaluates whether an agent can be manipulated into violating its intended security policies, permissions or trust boundaries. It includes testing prompts, memory, retrieval, tools, MCP integrations, identity, authorization, data movement and external actions.

How is AI agent security different from LLM security?

LLM security primarily focuses on model interactions and outputs. AI agent security also covers the systems surrounding the model, including tools, memory, retrieval, permissions, external APIs and actions.

Can AI agents leak their system prompts?

Agents may reveal or allow inference of parts of hidden instructions depending on their architecture and safeguards. Security testing should therefore measure both verbatim prompt leakage and semantic reconstruction of hidden policies.

Can an AI agent be reverse engineered?

Parts of an agent may sometimes be inferred through its observable behavior, but complete reconstruction is not guaranteed. The relevant security question is how much proprietary or sensitive information can be recovered under a defined attacker-access model.

What is Agent Extractability?

Agent Extractability measures how much proprietary, sensitive or security-relevant information about an AI agent can be reconstructed through the interfaces available to an attacker.

What is AI agent memory poisoning?

Memory poisoning occurs when malicious or misleading information enters an agent's persistent state and influences later reasoning or actions.

What is MCP tool poisoning?

MCP tool poisoning occurs when malicious tool information or responses influence an agent's behavior beyond the intended purpose of the tool.

What is an AI agent trust boundary?

An AI agent trust boundary is a point where data, authority or state moves between contexts with different levels of trust, such as between users, sessions, agents, tools or tenants.

What is an AI agent firewall?

An AI agent firewall is a runtime security layer that evaluates agent interactions and actions against security policies before allowing sensitive operations to execute.

How often should AI agents be security tested?

Agents should be tested before production and retested after meaningful changes to models, prompts, tools, memory, retrieval systems, policies or external integrations. High-risk production agents also benefit from continuous monitoring and regression testing.

---

The Bigger Security Question

For years, application security focused heavily on whether an attacker could break into a system.

Agentic systems introduce another question:

Can an attacker make the system legitimately use its own capabilities against the security intent of its designers?

An agent may have valid credentials.

It may call a legitimate tool.

The tool call may contain valid parameters.

And the resulting action can still be wrong.

That is why securing agents requires more than protecting the LLM.

You need to understand:

who supplied the information,

where that information traveled,

what authority the agent inherited,

which tools became reachable,

and

whether the resulting action was actually permitted.

The future of AI agent security will therefore be less about collecting increasingly clever jailbreak prompts and more about continuously testing the boundaries between:

identity, context, state, memory, models, tools and actions.

The most important question is no longer:

“Did the model say something unsafe?”

It is:

“Did the agent cross a boundary it was never supposed to cross?”