An agent eval is an automated test that tells you whether your AI agent got better or worse after a change. You run the agent on a fixed set of real tasks, record everything it did, and score both the final answer and the steps it took to get there. Evals solve one problem: agents are not deterministic, so you can't tell by looking whether a new prompt, model or tool broke something. The most common mistake is to grade only the final answer. Agents usually fail in the middle of a task, so a useful eval checks the path as well as the result.
What problem do agent evals solve?
Normal software gives the same output for the same input, so a unit test stays true until somebody changes the code.
Agents don't work like that. The same request can produce a different answer or a different sequence of tool calls on each run. And the agent changes even when you don't touch your code:
• You edit the system prompt to fix one customer complaint.
• The model provider releases a new version, or retires the one you use.
• You add a tool, or a tool's API starts returning a new field.
• Your data changes, so retrieval returns different documents.
Each of these can fix one case and quietly break ten others. Without evals, you only find out when a user complains, and by then you don't know which change caused it.
Evals turn "I think the new prompt is better" into a number you can compare with last week's. They also answer business questions that are otherwise guesses, like whether a cheaper model is good enough.
What types of agent evals exist?
There are three useful ways to divide evals: by what you grade, who does the grading, and when the eval runs.
What you grade: outcome vs. path
Outcome evals check the end state. Is the answer correct? Was the ticket updated with the right status? This is the most important check, but it's not enough on its own.
Path evals (also called trajectory evals) check how the agent got there: which tools it called, in what order, how many steps it took, and how many tokens and seconds it used.
An agent can reach the right answer by a fragile route: wrong tool first, an error, a retry, and success on the third attempt. An outcome eval scores that as a pass. A path eval shows you a failure before it happens.
Who grades: code, an LLM judge, or a human
Code-based graders are ordinary asserts. Is the output valid JSON? Was `get_customer` called before `update_customer`? Did the run finish in under 8 steps? They are fast, cheap and give the same result every time. Use them for everything they can check, which is more than most teams expect.
LLM-as-a-judge graders use a second model to score what code can't check: did the summary leave out an important fact, did the agent answer the question the user actually asked. They are useful, but a judge is also a model and can be wrong in consistent ways, so it needs its own testing.
Human review is the most accurate and the most expensive. Use it to build the first eval set, to check the LLM judge, and to settle cases where graders disagree.
When it runs: offline vs. online
Offline evals run before deploy, on a fixed dataset, usually in CI. They answer: "Is this change safe to ship?"
Online evals score a sample of live production traces with the same graders. They answer: "Is the agent still working with real users and real data?"
You need both. Offline evals catch regressions you cause. Online evals catch the ones the world causes.
How and when do evals work?
Evals run as a loop:
1. Dataset. A set of test cases. Each case is an input (a user request, plus any starting state such as a customer record) and what a good result looks like.
2. Run. The agent runs each case in an environment where tools are real or realistically mocked.
3. Trace. Every step is recorded: the model calls, tool calls, arguments, results, tokens, and time.
4. Grade. Graders score the outcome and the path for each case.
5. Compare. Scores are compared against the last accepted version, not against a perfect score.
6. Decide. If quality drops below the agreed level on any important category, the change doesn't ship.
When to run them:
• On every change to prompts, tools or agent code. Run the fast, code-graded part in CI on each pull request.
• Before every model change, including "minor" version updates. Run the full set.
• On a schedule, nightly or weekly, because providers change behaviour without you changing any code.
• Continuously in production. Score a sample of live traces and send low scores to a person.
Best practices for agent evals
Start with 20 to 50 real cases, not 1,000 invented ones. Take them from real user requests, support tickets or logs. A small set you have read carefully is worth more than a large generated set that nobody has checked.
Turn every production failure into a test case. When something goes wrong, add that input to the eval set before you fix it. Over time, the set becomes a record of how your agent actually fails, not how you guessed it would.
Grade the path with code first. Tool order, step count, no write before the read it depends on: each is a few lines of code. Use an LLM judge only where an assert can't do the job.
Test your LLM judge against human labels. Have a person label 50 to 100 cases, run the judge on the same cases, and measure how often they agree. If they agree less often than two humans do, fix the judge's prompt before you trust its scores.
Run each case more than once. An agent that solves a task on one run out of three is not reliable, even if one run is a pass. Run important cases several times and track how often the agent succeeds on every run, not only on at least one. Research benchmarks such as τ-bench call this pass^k, and scores drop sharply as k grows.
Treat cost and latency as quality metrics. A change that keeps accuracy the same but doubles the number of steps is a regression. Put tokens per task and time per task on the same report as accuracy.
Version the eval set like code. Keep it in the repository and record which version of the set produced which score, so this month's number can be compared with last month's.
What evals can't do
Evals don't prove the agent works. They prove it works on the cases you thought to include.
The dataset gets old. If nobody adds cases from production, the eval set describes last quarter's users.
LLM judges often prefer longer answers and change behaviour when their own model is updated. Pin the judge's model version and recheck it against human labels from time to time.
And don't aim for 100%. Some cases are ambiguous, and some failures are acceptable if a human catches them. Agree with the business which categories must never fail (refunds, deletions, anything that touches money or customer data) and set strict thresholds there. Be more relaxed about the rest.
Where to start this week
If you have an agent in production or close to it and no evals yet:
1. Export 30 real conversations or traces, including the ones that went wrong.
2. For each one, write down what a correct result is and which tool calls should have happened.
3. Write code checks for those tool calls and for the final state.
4. Run the set against your current version and save the score as the baseline.
5. Add it to CI, so no prompt or model change ships without a comparison.
That's a few days of work. After that, every change to your agent comes with a number instead of a guess.
If you want help setting this up on an agent that already has real users, we do this as part of our AI agent development work. You can see how it worked on a live system in the Mozi case.
Book a fit call — 30 minutes. We'll look at your agent and tell you where an eval set would catch the most problems first.