How to Evaluate AI Agents (Not Just Models)

Evaluating a plain model is straightforward: send a prompt, grade the answer. Agents break that simplicity completely. An agent takes many steps, calls tools, reads results, changes plans, and might reach a good answer through a terrible path — or fail a task it almost solved. Scoring only the final answer misses most of what determines whether an agent is actually good.

The shift is from grading an output to grading a trajectory. You care not just whether the agent finished, but how: which tools it used, how many steps it wasted, whether it noticed and fixed its own errors, and what it cost. Evaluating agents well is harder than evaluating models, and skipping it is how teams ship agents that demo perfectly and fail in production.

Why model evals are not enough

An agent is a loop, not a function, and that changes what can go wrong. The same final answer can come from a clean two-step path or a flailing twelve-step one, and the difference matters for cost, latency, and reliability.

• Outcome is not the whole story. An agent can luck into the right answer through a broken process that will fail on the next task.
• The path has its own failures. Wrong tool, wrong arguments, redundant searches, infinite loops — none of these show up in the final answer alone.
• Runs are non-deterministic. The same agent on the same task can succeed once and fail the next time, so a single run tells you little.
• Failure is hard to attribute. When a ten-step run goes wrong, you need to know which step broke it, not just that it broke.

The dimensions that matter

Good agent evaluation looks at several axes at once. Outcome: did it actually complete the task, measured by a checkable success condition. Process: did it take a sound path — valid tool calls, reasonable steps, no loops. Efficiency: how many steps, how many tokens, how much wall-clock and cost it spent getting there.

Then resilience: when a tool errored or returned junk, did the agent notice and recover, or did it barrel ahead? And safety: did it stay within its allowed actions and avoid doing something destructive. A strong agent scores well across all of these; a fragile one often hits the outcome while failing the rest.

Scoring a run, in code

In practice you run the agent over a suite of tasks with checkable goals and record more than success — the path and its cost too:

# Evaluate an agent run: did it reach the goal, and was the path sound?
def eval_agent(task, run):
    return {
        "success": task.goal_met(run.final_answer),    # outcome: did it finish the task?
        "steps": len(run.steps),                        # efficiency: how many actions it took
        "valid_tool_calls": all(s.tool_ok for s in run.steps),   # process: valid tool use?
        "recovered": any(s.was_error and s.fixed for s in run.steps),  # resilience after errors
    }
# Run the whole suite multiple times; track success RATE and efficiency, not one lucky example.

For the subjective parts — was this the right plan, was that step reasonable — an LLM-as-judge over the trajectory works, with the same caution it needs anywhere: a clear rubric and calibration against human judgment.

Grading an agent by its final answer is like judging a road trip by whether the car arrived — ignoring that it ran three red lights and took the long way twice. The journey is the product.

Practices that keep evals honest

• Use checkable tasks. Build a suite where success can be verified automatically — a file written, a value returned, a test passed — not just vibes.
• Run each task many times. Non-determinism means success rate over repeats, not a single pass, is the real signal.
• Inspect trajectories, not just scores. Trace the steps so you can see where and why a run failed, and catch right-answer-wrong-path successes.
• Measure cost alongside success. An agent that succeeds in twenty steps is worse than one that succeeds in three; track steps, tokens, and latency.
• Test failure injection. Make tools error or return bad data on purpose and check whether the agent recovers — resilience is where production agents live or die.

The bottom line

Agents are loops with tools, so evaluate the whole run, not just the last line. Score outcome, process, efficiency, resilience, and safety; use checkable task suites; repeat each task to beat non-determinism; and read the trajectories to see what actually happened. That is how you tell a genuinely reliable agent from one that got lucky in the demo.

The teams whose agents survive contact with real users are the ones who measure the journey, not just the destination. Build that evaluation harness early — before you ship — and let it tell you which agent to trust.