AI Agent Evaluation: How to Test Tool Use, Decisions, and Multi-Step Workflows

A convincing final answer does not prove that an AI agent behaved correctly.

An agent can reach the right result through the wrong tool, use incorrect arguments that happen to work, skip required approval, repeat an irreversible action, lose state, recover badly from a tool failure, or continue acting after the task is already complete.

That makes AI agent evaluation different from ordinary output evaluation.

INPUT
  ↓
OUTPUT
  ↓
SCORE

A tool-using agent creates a much richer path:

GOAL
  ↓
DECISION
  ↓
TOOL SELECTION
  ↓
ARGUMENTS
  ↓
TOOL RESULT
  ↓
STATE CHANGE
  ↓
NEXT DECISION
  ↓
MORE ACTIONS
  ↓
FINAL OUTCOME

The final response is only one part of that trajectory.

A good agent is not only one that produces a convincing final answer. It is one that reaches the correct outcome through appropriate actions.

This guide explains how to evaluate both outcomes and trajectories, including task success, tool selection, tool arguments, action ordering, state transitions, grounding, recovery behavior, authorization boundaries, stop conditions, cost, latency, and unnecessary actions.

Quick Answer

An effective agent evaluation usually checks at least three levels:

1. OUTCOME
Did the task actually succeed?

2. TRAJECTORY
Did the agent use the right tools, arguments,
state transitions, and recovery path?

3. EFFICIENCY / BOUNDARIES
Did it avoid unnecessary work, respect approvals,
stop correctly, and stay within acceptable cost and latency?

Do not evaluate only the final answer when the agent has authority to take actions.

Key Takeaways

• Agent evaluation should inspect both the final outcome and the path used to reach it.
• Task success alone can hide bad tool selection, duplicate actions, invalid arguments, and unsafe sequencing.
• Tool selection and tool argument accuracy are separate evaluation dimensions.
• Sometimes the correct tool decision is to use no tool at all.
• Action ordering matters when workflows have prerequisites or irreversible actions.
• State assertions can catch agents that act before required conditions become true.
• Failure recovery should be tested intentionally.
• Authorization and approval boundaries should be explicit eval cases.
• Stop-condition testing catches agents that continue acting after success.
• Efficiency metrics such as tool-call count, retries, latency, tokens, and cost matter.
• Use deterministic checks whenever the expected condition can be verified exactly.
• Model-based grading is useful for nuanced quality but should not replace exact checks.
• Human evaluation remains valuable for ambiguous outcomes and domain judgment.
• Datasets should include normal, edge, failure, adversarial, and historical regression cases.
• Production failures should become future regression tests.
• PrompTessor can improve prompt instructions being evaluated, but it is not an agent evaluation runtime.

Table of Contents

• What Is AI Agent Evaluation?
• Output Evaluation vs. Agent Evaluation
• Outcome vs. Trajectory
• Anatomy of an Agent Evaluation
• 1. Task Success
• 2. Tool Selection
• 3. Tool Argument Accuracy
• 4. Action Sequence
• 5. State Management
• 6. Evidence and Grounding
• 7. Error Recovery
• 8. Safety and Authorization
• 9. Stop Conditions
• 10. Efficiency
• Turn-Level, Trajectory-Level, and End-to-End Evals
• Deterministic, Model-Based, and Human Graders
• How to Build an Agent Evaluation Dataset
• Failure Injection
• Agent Regression Testing
• Agent Evaluation Metrics
• Current Evaluation and Tracing Patterns
• Practical Agent Evaluation Examples
• Common Agent Evaluation Mistakes
• Where PrompTessor Fits
• Agent Evaluation Checklist
• Related PrompTessor Guides
• Official Resources
• FAQ

What Is AI Agent Evaluation?

AI agent evaluation is the process of testing whether an agent completes tasks correctly and whether its actions during the task are acceptable.

For a tool-using agent, evaluation can include the final outcome, selected tools, tool arguments, action order, state changes, handoffs, retrieved evidence, approval behavior, error recovery, stop behavior, latency, token usage, and cost.

This extends the ideas in AI Prompt Evaluation. Prompt evaluation asks whether an instruction reliably produces the desired output. Agent evaluation adds the question of whether a multi-step system made the right decisions and actions along the way.

Output Evaluation vs. Agent Evaluation

Output Evaluation	Agent Evaluation
Focuses on returned content	Focuses on outcome plus actions
Often one model call	Often multiple model and tool calls
Checks correctness, relevance, format	Also checks tools, arguments, state, order, recovery
May not need environment state	Often depends on environment state
Can grade one answer	May need the full trajectory

Outcome vs. Trajectory

Two agents can reach the same visible outcome with very different quality.

AGENT A
check availability → create event → done
2 tool calls

AGENT B
check availability → search again → unrelated tool
→ retry → create event → create event again → done
6 tool calls

If both return “Your meeting has been scheduled,” a final-answer-only evaluator may treat them as equivalent. They are not. Agent B may have created a duplicate event, incurred higher cost, increased latency, and demonstrated poor stopping behavior.

Evaluate the outcome and the path used to reach it.

Anatomy of an Agent Evaluation

1. Task Success
2. Tool Selection
3. Tool Argument Accuracy
4. Action Sequence
5. State Management
6. Evidence / Grounding
7. Error Recovery
8. Safety / Authorization
9. Stop Conditions
10. Efficiency

Not every agent needs every dimension. Choose the dimensions that reflect what can actually go wrong in the workflow.

1. Task Success

Task success asks whether the user's goal was actually achieved.

TASK
Schedule a 30-minute meeting with Sarah next week.

SUCCESS
- correct Sarah
- valid date next week
- 30-minute duration
- no availability conflict
- correct timezone
- event exists exactly once

When possible, verify success from actual environment state rather than trusting the agent's claim that the action completed.

2. Tool Selection

Tool selection asks whether the agent chose the right capability for the current decision.

AVAILABLE TOOLS
search_internal_docs
search_web
get_customer
refund_order
send_email

For the task “What is our enterprise refund policy?”, a likely correct choice is search_internal_docs, not refund_order.

Include no-tool cases too. If trusted context already contains the answer, an unnecessary tool call adds latency, cost, and new failure modes.

For the instruction-design side, see Function Calling and Tool Use.

3. Tool Argument Accuracy

Choosing the correct tool is not enough.

{
  "tool": "create_event",
  "start_time": "2026-09-12T10:00:00+08:00",
  "timezone": "Asia/Makassar",
  "attendee": "[email protected]",
  "duration_minutes": 30
}

Check required fields, values, normalized dates, timezones, IDs, enums, units, and whether the model invented unavailable values.

RIGHT TOOL + WRONG ARGUMENTS = FAILURE

4. Action Sequence

Many workflows have ordering constraints.

identify customer
↓
load order
↓
check eligibility
↓
obtain approval
↓
issue refund
↓
verify result

Evaluation can classify steps as required, forbidden, ordered, optional, or conditionally required. This is especially important for irreversible actions.

5. State Management

customer_verified = true
refund_eligible = true
approval_received = false
refund_executed = false

The agent should not issue a refund while approval_received = false.

State evaluation can assert transitions and block actions that occur before prerequisites become true.

See AI Agent Memory and State Management for the distinction between current state, history, working memory, and long-term memory.

6. Evidence and Grounding

Research and RAG agents should be evaluated on whether their claims are supported by the evidence they actually retrieved.

• Did the agent retrieve relevant evidence?
• Did the final claim exist in the evidence?
• Did it distinguish missing evidence from negative evidence?
• Did it handle source conflicts?
• Did citations point to supporting sources?
• Did untrusted content alter instruction authority?

See RAG Prompting and Prompt Injection.

7. Error Recovery

Production tools fail, so evals should intentionally test timeouts, HTTP errors, empty results, rejected arguments, expired authorization, stale resources, and partial success.

TOOL FAILURE
↓
classify failure
↓
retry if policy allows
↓
verify result
↓
if unresolved: stop safely and report failure

A severe failure mode is when the tool fails and the agent still tells the user the action succeeded.

8. Safety and Authorization

READ EMAIL     allowed
DRAFT EMAIL    allowed
SEND EMAIL     requires approval

Create eval cases where the action is ready but approval is absent. The correct result may be to ask for confirmation rather than act.

For designing these boundaries in agent instructions, see AI Agent Prompts.

9. Stop Conditions

A reliable agent must know when the job is done.

task complete
↓
search again
↓
call another tool
↓
modify another resource

Useful metrics include post-success actions, duplicate writes, repeated searches, unbounded retries, and unnecessary handoffs.

10. Efficiency

Metric	Agent A	Agent B
Task success	97%	97%
Average tool calls	4.2	11.8
Median latency	5.4s	14.1s
Average cost	$0.018	$0.061
Duplicate writes	0.1%	1.8%

Agent A is likely a stronger production candidate despite identical task success. Efficiency should be measured relative to required verification and safety, not by minimizing every step blindly.

Turn-Level, Trajectory-Level, and End-to-End Evals

Turn-Level

Evaluate one decision in isolation, such as which tool should be used next given a known state.

Trajectory-Level

Evaluate a sequence of decisions, tools, retries, state changes, and handoffs.

End-to-End

Run the agent in a realistic test or sandbox environment and verify the final state.

A mature suite often combines all three because they trade off speed, realism, and debuggability.

Deterministic, Model-Based, and Human Graders

Deterministic Checks

Use for exact conditions such as tool names, arguments, schema validity, required steps, forbidden actions, final state, action counts, latency, and cost.

Model-Based Graders

Use for nuanced criteria such as completeness, clarity, semantic task success, or explanation quality when exact matching is inappropriate.

Human Review

Use for ambiguous goals, high-risk edge cases, subjective UX quality, domain expert judgment, and grader calibration.

DETERMINISTIC CHECKS
+
MODEL-BASED GRADERS
+
TARGETED HUMAN REVIEW

How to Build an Agent Evaluation Dataset

EVAL DATASET
=
NORMAL
+
EDGE
+
FAILURE
+
ADVERSARIAL
+
HISTORICAL REGRESSION

Normal cases establish baseline capability. Edge cases test ambiguity and unusual input. Failure cases test tool and environment recovery. Adversarial cases test trust and authorization boundaries. Historical regressions keep known failures from returning.

Failure Injection

Do not wait for external services to fail naturally during evaluation.

CASE: calendar timeout

first get_availability call → timeout
second call → success

Expected behavior may include recognizing the timeout, retrying only according to policy, avoiding duplicate writes, and continuing after a successful retry.

Another useful case is a write that succeeds while the confirmation response is lost. The agent should verify whether the write already happened before retrying.

Agent Regression Testing

AGENT PROMPT v4
success              93%
tool calls           4.1/task
approval failures    0

AGENT PROMPT v5
success              95%
tool calls           7.8/task
approval failures    2

v5 improved task success but introduced authorization failures and higher tool usage. That is not automatically a better release.

See Prompt Refinement for targeted iteration and AI Prompt Evaluation for reusable test-set design.

Agent Evaluation Metrics

Category	Example Metrics
Outcome	Task success rate, final-state accuracy
Tool Use	Tool-selection accuracy, no-tool accuracy
Arguments	Required-field accuracy, exact-value accuracy, schema validity
Trajectory	Required-step coverage, forbidden-action rate, ordering violations
State	Invalid transitions, stale-state actions, duplicate actions
Grounding	Supported-claim rate, citation precision, hallucination rate
Recovery	Recovery success, retry compliance, false-success rate
Authorization	Unauthorized-action rate, approval compliance
Stopping	Post-success actions, repeated calls, loop rate
Efficiency	Tool calls, tokens, latency, retries, cost per successful task

Avoid collapsing everything into one weighted score too early. A high average can hide a zero-tolerance failure such as sending without approval.

Current Evaluation and Tracing Patterns

OpenAI

OpenAI's current Agents SDK includes built-in tracing for events such as LLM generations, tool calls, handoffs, guardrails, and custom events. Its current testing documentation also describes deterministic provider-neutral test doubles for agent workflows. OpenAI's Evals API supports evaluation runs with configurable criteria, and its Graders API includes deterministic and model-based grader types.

Google Agents CLI

Google's current evaluation guide includes agent-oriented metrics for tool-use quality, multi-turn tool use, trajectory quality, task success, grounding, hallucination, and safety. It also supports generating and grading traces separately.

Agent evals work best when the evaluation system can observe the trajectory, not only the final message.

Practical Agent Evaluation Examples

These examples cover distinct failure patterns. The count is based on topic coverage, not a fixed article template.

Example 1: Calendar Agent

Task: Schedule a meeting with a saved contact

What to evaluate: Correct contact, availability check, timezone, one event, no duplicate, correct stop

Example 2: Customer Support Agent

Task: Resolve a refund request

What to evaluate: Eligibility check, approval boundary, correct refund action, accurate customer response

Example 3: RAG Research Agent

Task: Answer from internal policy documents

What to evaluate: Relevant retrieval, grounded claims, source conflict handling, no injected instruction following

Example 4: Email Agent

Task: Draft and send an outreach email

What to evaluate: Correct recipient, draft quality, no send before approval, send exactly once after approval

Example 5: Coding Agent

Task: Fix a failing test

What to evaluate: Correct files changed, tests executed, no unrelated modifications, recovery from failed command

Example 6: Browser Agent

Task: Complete a multi-page web workflow

What to evaluate: Correct navigation, form values, no repeated submission, handle changed page state

Example 7: Shopping Agent

Task: Find products under a budget

What to evaluate: Constraint adherence, current product data, no purchase without authorization

Example 8: Data Analysis Agent

Task: Analyze uploaded data and create findings

What to evaluate: Correct tool use, valid calculations, traceable results, no unsupported conclusions

Example 9: Automation Agent

Task: Run a recurring operational workflow

What to evaluate: Correct preconditions, action order, idempotency, retry behavior, stop condition

Common Agent Evaluation Mistakes

1. Evaluating Only the Final Answer

The final message can hide incorrect tools, duplicate actions, or unsafe paths.

2. Testing Only Happy Paths

Agents need cases where context is missing, ambiguous, stale, or broken.

3. Checking Tool Name but Not Arguments

The correct function with the wrong ID, amount, date, or timezone can still fail.

4. Ignoring Unnecessary Tool Calls

Extra actions increase cost, latency, and risk.

5. No State Assertions

Without state checks, agents can act before prerequisites are satisfied.

6. No Failure Injection

Recovery behavior remains untested until production.

7. No Authorization Tests

Approval boundaries should be explicit pass/fail conditions.

8. Using Only LLM-as-Judge

Do not use a model to judge values that can be checked exactly.

9. No Historical Regression Cases

Past failures should become future tests.

10. Comparing Versions Under Different Conditions

Different models, tools, or environment state can make comparisons misleading.

11. Ignoring Cost and Latency

A successful agent can still be operationally poor.

12. No Stop-Condition Tests

Agents that keep acting after success can create side effects.

Where PrompTessor Fits

PrompTessor fits in the prompt iteration side of an agent evaluation loop.

AGENT INSTRUCTIONS
      ↓
PrompTessor
├ Generate
├ Analyze
├ Optimize
├ Refine
└ Compare prompt iterations
      ↓
AGENT PROMPT CANDIDATE
      ↓
YOUR AGENT RUNTIME
      ↓
YOUR EVALUATION HARNESS
├ task success
├ tools
├ arguments
├ state
├ authorization
├ recovery
├ stopping
├ latency
└ cost
      ↓
FAILURES / RESULTS
      ↓
REFINE THE PROMPT AGAIN

PrompTessor's current AI Prompt Analysis documentation recommends validating prompts before repeatable or production use. Its Prompt Optimizer workflow recommends testing optimized versions in the target AI tool and comparing them with the original. Prompt Refinement supports feedback-driven iteration before prompts are reused.

PrompTessor does not execute agent evaluation suites, simulate tool environments, automatically grade trajectories, run production traces, allocate test traffic, or provide a full agent observability platform.

For instruction design, see AI Agent Prompts. For tool policies, see Function Calling and Tool Use. For stateful agents, see AI Agent Memory and State Management.

Agent Evaluation Checklist

• Is task success defined in terms of actual environment state?
• Can the final outcome be checked independently from the agent's claim?
• Are expected tools known for each case?
• Are no-tool cases included?
• Are tool arguments checked separately?
• Are required and forbidden steps documented?
• Are ordering constraints tested?
• Are important state transitions asserted?
• Are grounding checks included where relevant?
• Are tool failures injected intentionally?
• Is retry behavior bounded?
• Are false-success claims tested?
• Are approval boundaries explicit?
• Are cross-user or cross-tenant boundaries tested where relevant?
• Are stop conditions tested?
• Are duplicate writes detected?
• Are tool-call counts recorded?
• Are latency and cost measured?
• Are normal, edge, adversarial, and regression cases included?
• Are deterministic graders used where possible?
• Are model graders calibrated?
• Is human review used for ambiguous or high-risk cases?
• Are versions compared under equivalent conditions?
• Can failures be traced to exact prompt/model/tool configuration?
• Do failed cases feed back into prompt refinement?

Related PrompTessor Guides

• AI Agent Prompts: How to Write Better Instructions for Tool-Using AI Agents
• Function Calling and Tool Use: How to Write Better Prompts for AI Tools
• AI Agent Memory and State Management
• AI Prompt Evaluation
• Prompt Injection
• RAG Prompting
• Context Engineering
• Structured Outputs
• Prompt Chaining
• Prompt Templates and Variables
• Prompt Refinement
• AI Prompt Analysis

Official Resources

• OpenAI Agents SDK — Tracing
• OpenAI Agents SDK — Testing
• OpenAI API — Evals
• OpenAI API — Graders
• Google Agents CLI — Evaluation Guide

FAQ

What is AI agent evaluation?

AI agent evaluation tests whether an agent completes its goal correctly and whether its tools, arguments, state transitions, recovery, authorization, and stopping behavior are acceptable.

How is agent evaluation different from prompt evaluation?

Prompt evaluation often focuses on outputs. Agent evaluation also examines multi-step actions, tool calls, environment state, and trajectories.

What is an agent trajectory?

The sequence of decisions, model calls, tool calls, results, state changes, handoffs, retries, and actions during a task.

Should I evaluate the final answer or the trajectory?

Both. Final outcomes matter, but trajectory evaluation reveals unsafe or inefficient behavior hidden by plausible responses.

What is task success for an AI agent?

The user goal was actually achieved in the target environment, not merely claimed by the agent.

How do you evaluate tool selection?

Define appropriate, inappropriate, and unnecessary tools for each case and compare the agent decision with those expectations.

Should no-tool behavior be evaluated?

Yes. Unnecessary tool calls can add cost, latency, and risk.

How do you evaluate tool arguments?

Validate required fields, IDs, values, dates, timezones, enums, units, and whether values were invented.

Why does action order matter?

Many workflows have prerequisites. Irreversible actions before verification or approval can cause serious errors.

How do you evaluate agent state?

Assert that important state values change correctly and actions occur only when required conditions are true.

How do you evaluate error recovery?

Inject controlled failures and test retry rules, verification, fallback behavior, and false-success handling.

How do you evaluate approval boundaries?

Create cases where an action is ready but approval is absent and verify that the agent does not execute it.

What is stop-condition evaluation?

Testing whether the agent stops after success and avoids duplicate writes, extra searches, or unbounded retries.

What efficiency metrics matter?

Tool calls, retries, latency, tokens, cost per successful task, duplicate actions, and unnecessary steps.

What is a deterministic grader?

Exact logic that checks conditions such as tool names, arguments, schemas, state, action counts, or final values.

When should I use an LLM judge?

For nuanced criteria such as completeness, semantic success, usefulness, or explanation quality when exact checks are insufficient.

Do I still need human evaluation?

Yes for ambiguous tasks, high-risk edge cases, domain expertise, UX quality, and grader calibration.

What should an agent eval dataset contain?

Normal cases, edge cases, controlled failures, adversarial cases, and historical regressions.

What is failure injection?

Deliberately making tools or environment components fail during tests to evaluate recovery before production.

What is agent regression testing?

Rerunning a stable evaluation suite after prompt, model, tool, or workflow changes to detect behavior that became worse.

Can two agents have the same success rate but different quality?

Yes. One may use more calls, cost more, violate approvals, or create duplicate actions despite reaching the same result.

How can PrompTessor help with agent evaluation?

PrompTessor can help generate, analyze, optimize, refine, and compare the prompt instructions being evaluated. The runtime, traces, graders, environment simulation, and observability remain outside PrompTessor.

Conclusion

AI agent evaluation requires a broader definition of quality than final-answer correctness.

A production agent should accomplish the intended task, select appropriate tools, produce valid arguments, follow required action order, preserve state, use evidence correctly, recover from failures, respect authorization boundaries, stop at the right time, and do all of that with acceptable cost and latency.

Evaluate the outcome and the path used to reach it.

Start with a small set of high-value cases. Add deterministic assertions wherever possible. Add model-based graders for criteria that require semantic judgment. Use targeted human review for ambiguity and high-risk behavior. Then preserve every meaningful failure as a regression test.

Over time, the evaluation suite becomes more than a scorecard. It becomes the specification for what reliable agent behavior actually means.