Evaluating AI Agents: Score the Trajectory, Not the Last Line

Here is the trap almost every team walks into with agents. You build an eval the way you always have: a set of prompts, a set of expected answers, a score. It works for models. You reach for the same tool for your agent, watch the pass rate climb to a comfortable number, and ship. Two weeks later the agent is quietly burning tokens in production, looping on a flaky API, and occasionally doing something you never sanctioned — and your eval said everything was fine. The eval was not wrong. It was answering a different question than the one you needed answered.

The mental model: a function vs. a flaky integration test

A plain model is a pure function. Prompt in, answer out. You grade it the way you grade a unit test: assert output equals expected. Deterministic-ish, cheap, one number.

An agent is not a function. It is a loop that reaches out and touches the world — it calls tools, reads what comes back, changes its mind, retries, and eventually stops. That is not a unit test. That is a flaky, stateful integration test that talks to live dependencies. And nobody sane grades an integration test by only checking the final return value while ignoring that it hammered the database 40 times and swallowed three exceptions on the way. So the shift is not subtle: you stop grading an output and start grading a trajectory — the ordered sequence of what the agent did, why, at what cost, and whether it stayed inside the lines. The final answer is one field on that record, not the record.

Grading an agent by its final answer is like judging a surgeon by whether the patient woke up — without checking whether they operated on the correct leg.

Once you hold that model, the failure modes that model evals cannot see become obvious. Same answer, wildly different path. A wrong tool call that got papered over. A retry loop that only stopped because it hit the step limit. A "success" that was one flipped coin away from failure. None of these are visible in the last line of output.

A worked example: the invoice agent

Say you have an agent whose job is: "Find the total on invoice #4471 and record it in the ledger. " It has three tools — search_docs, read_pdf, and write_ledger. Two runs both end with the ledger showing the right number. A model-style eval scores both a clean pass. Watch what the trajectories actually contain.

Run A (the good one, 3 steps): search_docs("invoice 4471") returns one hit. read_pdf on that hit returns the total. write_ledger records it. Done — every call valid, no waste, nothing touched outside its toolbox.

Run B (the lucky one, 9 steps): search_docs returns three hits because the query was vague. The agent reads the wrong PDF first (invoice #4171), gets a total, and tries to write it. write_ledger errors because the invoice ID does not match the row. The agent does not notice the mismatch — it retries write_ledger with the same bad value, fails again, re-searches, reads a second wrong PDF, fails once more, and only on the third read lands on #4471 and writes the correct total.

Same final ledger state. Same "pass. " But Run B took three times the steps, produced two failed writes, and — the part that should scare you — it did not recover because it reasoned about the mismatch. It recovered by brute force. The next time the target sits at hit #4 instead of hit #3, it runs out of steps and ships a wrong number with full confidence. Run A is a good agent; Run B is a coin flip wearing a green checkmark. A trajectory-aware eval separates them instantly. A final-answer eval calls them identical. That gap is the entire reason agent evaluation exists.

A scorecard, in code — and the number that catches the liars

The fix is to stop returning a boolean and start returning a scorecard. You still need a per-task goal check you write yourself — unavoidable, and the honest part of the work. But around it you capture the path, the cost, and two things the final answer can never tell you: did it recover from its own errors, and did it stay inside its permitted tools.

from dataclasses import dataclass, field

@dataclass
class RunTrace:
    task_id: str
    steps: list          # each: {"tool": str, "args": dict, "ok": bool}
    final: str
    tokens: int
    wall_ms: int

def grade(trace, goal_check, allowed_tools):
    """Return a scorecard, not a single number."""
    outcome = goal_check(trace.final)        # you write this per task
    calls = [s["tool"] for s in trace.steps]

    illegal = [t for t in calls if t not in allowed_tools]
    failed_calls = sum(1 for s in trace.steps if not s["ok"])
    # did a failed call get followed by a different, successful action?
    recovered = any(
        (not trace.steps[i]["ok"]) and trace.steps[i + 1]["ok"]
        for i in range(len(trace.steps) - 1)
    )

    return {
        "outcome": outcome,                  # bool: did it meet the goal
        "steps": len(trace.steps),
        "failed_calls": failed_calls,
        "recovered": recovered,
        "safe": len(illegal) == 0,           # stayed inside its toolbox
        "tokens": trace.tokens,
        "wall_ms": trace.wall_ms,
    }

The recovered heuristic just measures a failed call immediately followed by a successful one. Crude, but honest for a first pass — it flags Run B above as "recovered by retry" instead of pretending nothing went wrong. For the genuinely subjective calls (was that the right plan? was re-searching reasonable or flailing? ), an LLM-as-judge reading the whole trajectory is the standard tool, with the standard caveat: give it a written rubric and calibrate it against a handful of human-labeled runs first.

One run, though, is noise. Agents are non-deterministic, so a task that passes once may fail one time in four; the only honest headline is a pass rate over many repeats. And the single most useful number is not that pass rate — it is the gap between it and a stricter cousin.

# Run the SAME task N times, because one run tells you almost nothing.
def summarize(cards):
    n = len(cards)
    passed = sum(1 for c in cards if c["outcome"])
    return {
        "pass_rate": passed / n,                     # the headline number
        "clean_pass_rate": sum(                       # passed AND behaved
            1 for c in cards if c["outcome"] and c["safe"]
            and c["failed_calls"] == 0) / n,
        "median_steps": sorted(c["steps"] for c in cards)[n // 2],
        "p95_tokens": sorted(c["tokens"] for c in cards)[int(n * 0.95)],
    }

# A gap between pass_rate and clean_pass_rate is your early-warning light:
# the agent is passing by luck or by doing sketchy things on the way.

clean_pass_rate is pass rate with a conscience: it counts only runs that succeeded AND made no illegal tool calls AND had zero failed calls on the way. When pass_rate is 0.92 and clean_pass_rate is 0.55, your agent is passing, but a third of its wins look like Run B — lucky, wasteful, or slightly out of bounds. That gap is a leading indicator of production pain, and a raw pass rate alone hides it right up until a user hits the ugly path.

Where to spend the effort — and what still fools you

Trajectory evaluation is not free. It needs instrumented runs, checkable tasks, and repeats — real engineering time and a real token bill. Spend it where the loop does something that matters.

• Use full trajectory evals when: the agent takes irreversible actions (writes, payments, emails), tool cost or latency is a first-class concern, it runs unattended, or "right answer, wrong path" can hurt someone downstream.
• Skip most of it when: the agent is a single-tool wrapper (a model with a search box), a human reviews every output before it lands, or you are still prototyping and the tool surface changes daily — instrument once the shape stabilizes.

But even a good harness has sharp edges. These five are the ones that produce a green dashboard hiding a broken agent.

1. Judge contamination. If the same model family both runs the agent and judges the trajectory, it tends to rate its own reasoning generously. Use a different model as judge, or anchor every judge score against human labels — and re-check when you upgrade either model.
2. Seed leakage into the task suite. If your checkable tasks were written by looking at what the current agent does well, you are grading it against its own reflection. Write tasks from the spec and from real user logs, not from watching the agent succeed.
3. The flaky-task trap. When pass rate drops, half the time the agent did not get worse — a tool did. Log tool error rates separately from agent decisions, or you will spend a day "fixing" a prompt to compensate for a rate-limited API.
4. Cost-blind pass rates. An agent that solves a task in 20 steps is worse than one that solves it in 3, but a pass rate ranks them equal. Always report median steps and p95 tokens next to success, and treat a rising step count as a regression even when pass rate holds.
5. Golden-path overfitting. Teams tune against the tasks in the suite until the numbers look great, then meet an input the suite never covered. Deliberately inject failures — make a tool return junk or error on purpose — and keep a rotating holdout of tasks the agent has never been tuned on.

The bottom line

Agents are loops with tools, so the unit of evaluation is the run, not the last line it printed. Capture the trajectory, score outcome and process and efficiency and recovery and safety together, repeat every task enough times to beat non-determinism, and watch the gap between pass rate and clean pass rate like a warning light. Do that and you can tell, before you ship, the difference between an agent that is genuinely reliable and one that just got lucky in the demo — which is the only difference your users will ever feel.