How to test an AI agent before deploying it

Quick answer

Test an AI agent in five layers: the agent in isolation with controlled inputs, each tool integration separately, the handoffs between agents, the failure and edge cases, and the human gates. Work through each layer before connecting to real systems or live data. The goal is not to prove the agent works perfectly — it's to understand exactly where and how it fails before users do.

Why testing AI agents is different

Unit testing a function is straightforward: fixed input, expected output, pass or fail. Testing an AI agent is more complicated for three reasons.

Outputs are non-deterministic. The same input can produce different outputs across runs. Tests that check for exact string matches will break constantly. What you can test reliably: whether the output is in the right format, whether it contains the required fields, whether the reasoning is plausible given the input. Not: whether it says exactly the right words every time.

Tool calls have side effects. An agent that sends an email, writes to a database, or calls a payment API does something irreversible when it runs. Tests that let agents call live tools without controls can corrupt data, send real notifications, or incur real costs. Every tool call needs a test double — a mock that returns realistic responses without hitting production systems.

Failures cascade between agents. In a multi-agent system, an agent that returns malformed output doesn't just fail — it passes bad data to the next agent, which may produce bad output that passes to the next, and so on. The visible failure can be several steps removed from the actual cause. Testing agents in isolation before connecting them makes the source of failures obvious.

Layer 1: test the agent in isolation

Before connecting an agent to anything, test it with controlled inputs and mock tool responses.

What to test:

• Does the agent use the right tool when given a clear trigger?
• Does it produce output in the expected format (JSON, structured text, specific fields)?
• Does it handle ambiguous inputs — partial data, missing fields, unexpected formats?
• Does it stay within its defined role, or does it attempt actions outside its remit?

How to do it: Write a set of fixed test cases: one clean input that should work perfectly, one with missing data, one with edge-case values, one designed to confuse it (contradictory instructions, empty inputs). Run each case multiple times — non-deterministic systems need several passes to surface intermittent failures.

Use mock tool responses that simulate both success and failure. An agent that only gets tested against successful API responses will hit an unhandled real-world failure sooner or later.

Layer 2: test each tool integration

Every tool the agent uses — a CRM, a database, a notification service, an external API — needs to be tested independently from the agent itself.

What to test:

• Does the integration authenticate correctly?
• Does it handle rate limits and timeouts gracefully?
• Does it produce the data shape the agent expects?
• What happens when the tool returns an error, an empty result, or unexpected data?

Why separately: If you test the tool inside the agent, failures are ambiguous — you can't tell if the agent is misbehaving or the tool integration is broken. Isolating the tool lets you verify the plumbing before connecting the reasoning layer.

For tools that have destructive actions (write, update, delete, send), test against a sandbox or staging environment, not production. If no sandbox exists, mock the write endpoint and verify the agent constructs the correct payload.

Layer 3: test the handoffs between agents

In a multi-agent system, agents pass work to each other. The output of one agent becomes the input of the next. This is where integration failures hide.

What to test:

• Does Agent A's output match the format Agent B expects as input?
• What happens when Agent A produces incomplete or malformed output? Does Agent B fail gracefully or cascade the error?
• Does the right context travel between agents — or does something get dropped in transit?

How to do it: Test handoffs pairwise: take the actual output from Agent A (captured from an isolation test) and feed it directly to Agent B as input. You're testing the interface between them, not the full workflow. This makes it easy to pinpoint which agent is responsible when something breaks in a connected run.

Define the expected shape of every handoff in advance. If Agent A should hand off a structured object with specific fields, write that schema down before you start testing. Agents that produce the right general content but the wrong structure are a common failure mode.

Layer 4: test failure modes and edge cases

Most testing effort goes into the happy path. Most production failures come from edge cases. Test both.

Failure modes worth testing explicitly:

Empty or null inputs. What does the agent do when it receives nothing? A well-designed agent should produce a structured error or a clear "no action" output — not hallucinate a response.

Tool failure. When an API returns a 500, times out, or returns empty results — does the agent retry, give up, or hand off to a human? Test all three expected paths.

Contradictory instructions. If the agent receives inputs that conflict with its system prompt — a request to perform an action outside its role, for example — does it refuse cleanly, or does it attempt the action anyway?

Prompt injection via tool outputs. If a tool returns user-generated content (a CRM note, a support ticket, a document), a malicious input could attempt to override the agent's instructions. Test whether the agent treats tool outputs as data, not as instructions.

Repetition and loops. In multi-agent systems, test whether an agent can get into a loop — calling a tool repeatedly, re-requesting work that was already completed, or passing a task back to the agent that sent it.

Layer 5: test human gates

If your agent workflow includes human review checkpoints — and in most production systems it should — the gate logic needs to be tested as carefully as the agent logic.

What to test:

• Does the approval notification reach the right person with the right context?
• What happens when the reviewer approves? Does the next step trigger correctly?
• What happens when the reviewer rejects? Does the agent revise, or does the workflow stop?
• What happens when no one responds? Timeout behaviour is the most commonly skipped test — and the one that causes the most production incidents.

A gate that works on the happy path but hangs indefinitely when ignored will eventually block something critical. Define and test the timeout path explicitly before deploying.

Before you deploy: staging and shadow mode

Once all five layers pass, run the complete workflow in a controlled environment before it touches production.

Staging environment: a copy of your production setup with real integrations but non-production data. Tests the full system end-to-end without risk. The minimum bar before any agent goes live.

Shadow mode: the agent runs against real production data and produces real outputs — but those outputs are not acted on. A human reviews what the agent would have done for a defined period. This is the highest-confidence pre-deployment validation. Run shadow mode for one to two weeks on workflows that are high-stakes or irreversible.

Canary rollout: deploy to a small percentage of real traffic first. Monitor outputs closely before expanding. Useful for workflows with high volume and moderate stakes.

The right approach depends on the risk profile of the workflow. A budget variance monitor can go straight from staging to production. A payment approval agent should run shadow mode for at least a week before the gate is removed.

Design your AI agent system in Envelope

Envelope turns a plain-language description of your agent workflow into a complete structured design — roles, tools, handoffs, human review gates, and a spec you can test layer by layer before you build. Free to start.

Start designing →

Frequently asked questions

Can you automate tests for AI agents?

Partially. You can automate input/output format validation, tool integration tests, and gate trigger tests. What you cannot fully automate is evaluating whether an agent's reasoning is correct for a given input — that requires human review of a representative sample. Build automation around the structural and integration layers; reserve human review for output quality.

How many test cases do I need?

Start with one clean case, two or three edge cases, and one explicit failure case per agent. Run each case three to five times to surface non-deterministic failures. Add cases when you find real-world failures in staging — treat each production bug as a missing test case.

What's the difference between testing and evaluation?

Testing verifies behaviour: did the agent call the right tool, produce the right output format, handle the failure correctly? Evaluation assesses quality: was the output good, accurate, helpful? Both matter. Testing comes first and is more automatable. Evaluation is ongoing and requires human judgment or a purpose-built LLM evaluator.

Do I need a staging environment?

Yes, for any agent that writes to real systems. Running tests against production data with production tool access is how you accidentally send real emails, corrupt real records, or trigger real payments. If you can't build a full staging environment, at minimum mock all write endpoints and test destructive actions with a human in the loop.

What should I monitor after deployment?

Track tool call success rates, output format compliance, gate response times, and timeout frequency. If your gate reviews are taking longer than expected, investigate whether the outputs are unclear or the reviewer load is too high. Set alerts for tool failures — a broken integration will silently degrade agent performance before it fails completely.

How do I test a multi-agent system without running the whole workflow?

Test each agent in isolation first, then test pairs, then the full chain. Use real outputs captured from isolation tests as inputs for the next agent — this way you're testing the interface between agents without depending on the upstream agent to produce a clean output on every run. Only run the full workflow once individual agents and their handoffs are validated.