AI agent evaluation: a production test plan
Short answer: AI agent evaluation tests whether an agent completes a task correctly, uses tools within its authority, and leaves systems in an acceptable state. A good evaluation checks the final result and the actions taken to reach it. A fluent final answer can hide a duplicate payment, an unauthorized record change, or a failed handoff. That is why a polished demo is weak evidence for production readiness.
This guide is about evaluating agents that take multi-step actions. It shows how to design test cases, score behavior, and set release gates. For prompt and answer quality across language-model features, see our guide to LLM testing. Our guide to AI agent use cases that survive production helps decide whether an agent fits a workflow. Use these methods before expanding an agent’s access or assigning it consequential work.
Define success as a task and a state
Start with a real business task, its starting conditions, and a verifiable end state. “Answer the customer” is too vague. For example, find an order and verify refund eligibility. Then create one request and report the result. Each step can be checked.
For each AI agent test case, record:
- The user request and relevant context.
- The data and tools available to the agent.
- The allowed actions and actions that require approval.
- The expected system state when the task is complete.
- Acceptable alternatives, refusal conditions, and escalation rules.
Verify changes against the test environment or application state. Do not grade only the agent’s narrative. An agent can claim it completed work that it never performed, or report success after making the wrong change.
Evaluate the whole trajectory
An agent is the model plus its instructions, tools, orchestration, permissions, and state. Evaluate that complete system. Tests should use production tool schemas and approval rules. Testing the model alone says little about the deployed agent.
Capture a trace for every run: tool names, arguments, results, retries, approvals, and final state. Then score separate dimensions rather than collapsing them into a single “quality” number:
Task outcome. Did the system reach the correct state? Use deterministic checks where possible, such as a database row, a ticket status, or a generated file.
Tool correctness. Did it choose an allowed tool and supply valid arguments? Did it avoid duplicate or unnecessary calls?
Policy and permission adherence. Did it stay within the user’s access and the task’s authority? Did it ask for approval before a sensitive or irreversible action?
Recovery. When a tool timed out, returned incomplete data, or rejected an action, did the agent retry safely, choose a valid alternative, or stop and escalate?
User communication. Did its final message accurately describe what happened, including what remains incomplete?
This approach combines trace checks with checks on the resulting environment. Our AI agent development services apply task cases, trace review, and end-state checks to production workflows.
Build cases from real work and failure modes
Use historical requests, support escalations, and reviewed production incidents as the starting point. Remove or mask personal and sensitive information, then preserve the structure that made the task difficult.
Include routine cases, ambiguous requests, missing information, conflicting records, permission boundaries, unavailable tools, and actions that must be approved. Add cases where the safest outcome is to ask a question or stop. A test set made only of clean happy paths rewards an agent that guesses when it should pause.
For a first pass, a few dozen carefully reviewed cases can reveal obvious failure patterns. Expand coverage from observed failures. Do not assume a large synthetic dataset represents your users. Model behavior varies, so run important cases more than once. Report the outcome distribution, not only the best run.
Use the right grader for each claim
Prefer deterministic checks for facts that software can verify. Check whether a record changed and required fields are present. Validate tool arguments against a schema. Confirm each action occurred exactly once.
Use a rubric or model-based grader for qualities that require interpretation. Examples include whether an explanation is clear or a response addresses the request. Calibrate graders against human review and inspect disagreements. A judge model has blind spots. It is not ground truth.
Keep the trace beside every score. Aggregate pass rates identify a trend, but the individual runs explain why it happened. If an agent reaches the right end state by violating a permission rule, the overall result must still fail.
Set release gates around risk
Choose thresholds for each important dimension before comparing runs. A release gate might prohibit unauthorized writes in high-risk tests and duplicate actions. It can also set a minimum success rate for routine cases. Set thresholds according to error impact and human review availability.
Run the suite when you change the model, prompt, tools, retrieval, policy, or orchestration. Compare quality with latency, cost, retries, and human escalations. A change that raises completion while also increasing unsafe tool calls is not an improvement.
Keep production monitoring separate from pre-release evaluation. Offline cases test known risks; monitoring and review find new ones. When an issue is confirmed, create a privacy-safe reproduction. Add it as a regression case to help prevent recurrence.
Put irreversible actions behind explicit controls
Use read-only access during early pilots when it can answer the product question. Scope each write tool to the minimum action and data it needs. Require confirmation for consequential changes. Make operations idempotent where possible. Log enough for a reviewer to reconstruct each action.
In regulated settings, technical evaluation is only one part of responsible deployment. Map applicable obligations with qualified counsel and compliance leaders. Keep evaluation evidence, approval rules, and operating ownership aligned with that review.
A practical starting sequence
Choose one narrow task with a clear owner and a measurable outcome. Write down its allowed actions and escalation points. Assemble real, reviewed cases and create a safe test environment. Add checks for both the final state and the action trace. Run the baseline, fix the largest failure category, and repeat before expanding the agent’s authority.
The useful result is more than a benchmark score. It is evidence about what the system can do reliably and what it must refuse. It also shows where a person must approve or take over.
What should an AI agent evaluation suite contain?
An evaluation suite should represent the work the agent is expected to perform and the risks around that work. A single success metric cannot show whether the agent followed policy or reached the correct result by luck.
Start the agent evaluation framework with the highest-volume task. Add edge cases that could cause financial, operational, privacy, or customer harm. Include expected refusal and handoff cases. Label every case with its business impact, expected state, and grading method.
| Test category | Example case | What to verify |
|---|---|---|
| Routine completion | Resolve a request with complete account details | Correct final state and clear user response |
| Missing information | A request omits the account identifier | Agent asks for the missing field instead of guessing |
| Conflicting data | Two records show different addresses | Agent identifies the conflict and routes it for review |
| Permission boundary | User asks for another customer’s details | No protected data appears in tool results or response |
| Tool failure | A downstream system times out | Retry is bounded and does not duplicate the action |
| Approval requirement | A request would change a consequential record | Agent pauses until an authorized person approves |
This matrix gives product, engineering, operations, and compliance teams a shared definition of success. It also makes gaps visible before the first test run. If no one can agree on the expected outcome, the task is not ready for autonomous handling.
How do you score an agent without hiding failures?
Keep an AI agent evaluation scorecard with separate measures for task success, policy compliance, tool correctness, recovery, and communication. These agent evaluation metrics should be reported by task category and risk tier. A high average must not hide a serious failure in a smaller group.
For example, an agent might resolve 90 of 100 routine requests. It might also make one unauthorized write in ten high-risk cases. Reporting only a 90% average makes that boundary failure easy to miss.
Set critical safety checks as hard gates. A policy violation should fail the run even if the final answer looks correct. For softer qualities, define a rubric with clear examples of acceptable and unacceptable behavior. Have domain reviewers score a sample before relying on an automated judge.
Record the model version, prompt version, tool schemas, data snapshot, and configuration for each run. Without that context, a score change cannot be traced to a specific system change. Versioning also helps reproduce a production incident in the test environment.
How should you test tool calls and side effects?
Run actions in an isolated environment with synthetic or masked data. Stub external systems when a real call could create a payment, notify a customer, or change a live record. The test should observe the attempted action without causing a real-world side effect.
Check more than the tool name. This tool call evaluation should validate arguments, caller identity, returned records, and final state. Confirm the agent did not make an extra write after receiving a successful response.
Test retries explicitly. Simulate a timeout after the downstream service commits an action but before the agent receives confirmation. A safe system checks whether the action already happened before trying again. Idempotency keys and state checks can reduce duplicate work, but verify the behavior in the actual integration.
Test approval flows as their own cases. The agent should explain what needs approval, wait for the correct role, and continue only after approval arrives. A rejection should end the action cleanly. A timeout should not turn into implied consent.
How do you move from offline tests to production?
Offline evaluation measures known tasks under controlled conditions. It cannot cover every phrasing, tool outage, or operational change that appears after launch. Production AI agent testing combines these cases with a staged rollout, learning from real traffic without exposing users to unbounded risk.
Begin with read-only access or a shadow mode. In shadow mode, the agent prepares a proposed action while the existing process remains in control. Compare its recommendation with the human result, but do not execute the recommendation automatically.
Next, let the agent prepare low-impact actions for human approval. Review samples of successful and failed runs, including the full traces. Only automate a write after the team has reviewed its error patterns and documented a safe fallback.
During rollout, track completion, escalations, tool errors, repeated attempts, latency, and cost. Review cases where users correct the agent or abandon the workflow. These signals reveal problems absent from offline tests. For more monitoring practices, see LLM observability in production.
What are common mistakes in AI agent testing?
Testing only happy paths. A useful AI agent testing framework includes ambiguous requests, unavailable tools, permission limits, and valid reasons to refuse.
Scoring only the final answer. Verify the action trace and resulting state. A correct explanation cannot undo an unsafe tool call. Multi-step agent testing must include both.
Trusting a single run. Model behavior varies. Run important cases several times and inspect the range of outcomes.
Using synthetic cases as the only evidence. Synthetic data helps expand coverage, but real, reviewed tasks should anchor the suite.
Changing too many parts at once. If the model, prompt, tools, and retrieval all change together, scores cannot explain which change caused the result.
Treating every failure equally. A formatting issue and an unauthorized transaction need different release treatment. Grade by impact as well as frequency. This is central to AI agent safety testing.
Conclusion: treat evaluation as part of the agent
AI agent evaluation is an ongoing engineering practice, not a launch checklist. Define tasks with verifiable outcomes, inspect traces, enforce hard safety gates, and turn confirmed incidents into new cases. Expand access only when the evidence supports it.
The right question is not whether an agent can complete a demo. Ask which tasks it completes reliably, which actions remain under human control, and how the system fails when conditions change. A clear answer gives teams a safer path from prototype to production.
Frequently asked questions
What is an AI agent evaluation benchmark?
An AI agent evaluation benchmark is a set of representative tasks with clear graders and expected outcomes. It measures task completion, safe tool use, permission adherence, and resulting system state.
How is agent evaluation different from LLM testing?
LLM testing often scores responses or retrieval. Agent evaluation checks multi-step tool use, permissions, side effects, and recovery. It also verifies the intended outcome in the connected system.
What should an AI agent test case include?
Include the request, starting data, available tools, allowed actions, approval rules, expected end state, and conditions for refusal or escalation. Clear criteria make the result reproducible.
How many test cases do I need?
Start with a few dozen representative, reviewed cases for one narrow task. Add cases from incidents and coverage gaps. Risk and task diversity matter more than a large uncurated count.
Should an AI agent evaluation use an LLM judge?
Use an LLM judge for qualities that need interpretation, then calibrate it against domain reviewers. Prefer deterministic checks for state changes, permissions, schemas, and other verifiable facts.
How do you test an AI agent safely?
Use an isolated environment with synthetic or masked data. Stub consequential external actions, restrict credentials, and confirm that tests cannot write to production systems.
How often should teams run agent evaluations?
Run the relevant suite after changes to models, prompts, tools, retrieval, policies, or orchestration. Schedule checks for hosted model changes and review production incidents continuously.
What is a good agent evaluation metric?
Use task success alongside separate measures for policy compliance, tool correctness, recovery, latency, cost, and escalation. Set hard failure gates for high-impact safety violations.
Can agents be evaluated in production?
Yes, through monitoring, human review, and controlled rollout. Start with shadow or approval modes for consequential tasks, and use privacy-safe failures to improve offline tests.
When is an AI agent ready for production?
A scoped release needs predefined outcome and safety thresholds. Representative tests must meet them, and monitored fallbacks must be in place. An owner should handle incidents and ongoing review.
EpochC builds AI agent development services with evaluation and operational controls designed into the workflow. See the clinical multi-agent API case study or start a project.