LLM-as-a-Judge: The Grader Has Opinions, and One of Them Is About You

Here is the sentence that should scare you: your evaluation dashboard is green, so you ship, and the model is worse. Not broken-worse. Subtly-worse. The kind of worse that shows up three weeks later as a support ticket that says 'the assistant made up a refund policy. ' Your judge scored that answer 4.6 out of 5.

LLM-as-a-judge -- using a strong model to grade the output of another model -- is now the default way to evaluate anything open-ended: summaries, chatbot replies, RAG answers, agent traces. It is genuinely one of the highest-leverage tools in the stack. It is also the single most common source of false confidence I see in AI systems, and almost every failure comes from the same root mistake: people treat the judge like a ruler when it is actually a witness.

The mental model: a correlated instrument, not a ruler

A ruler has error, but the error is independent of what you are measuring. Measure a table twice and the millimeter you're off by has nothing to do with the table being wooden. An LLM judge is not like that. Its errors correlate with the very thing you are trying to measure. It rates fluent, confident, well-structured wrong answers highly -- precisely because fluency and confidence are what your model got better at producing. The judge's blind spots line up with your model's failure modes. That is the whole problem in one sentence.

So the right frame is not 'the judge gives me the score. ' It is 'the judge gives me a noisy signal that is systematically biased in a direction I can predict. ' Predictable bias is workable -- you correct for it. Unpredictable noise you average out. What kills you is pretending the bias isn't there.

Treat the judge as a witness, not a ruler. A witness can be sincere and specific and still be wrong in exactly the way that flatters your defendant. You cross-examine a witness. You do not just read its testimony into the record.

A worked example: where the naive judge gets it exactly backwards

Let's make this concrete. Say you have a RAG system that answers billing questions from a knowledge base. You're comparing model A (your current prod model) against model B (a candidate you want to ship). The question: 'Can I get a refund after 30 days? ' The knowledge base clearly says refunds are only available within 14 days.

Answer A (prod): 'Refunds are available within 14 days of purchase. After 14 days, purchases are non-refundable, though you can contact support to discuss exceptions. ' Hedged, correct, a little dry. Answer B (candidate): 'Great question! While our standard window is 14 days, customers who reach out after 30 days can often still qualify for a full refund under our satisfaction guarantee -- just contact support and mention your order number. ' Warm, confident, beautifully structured -- and it invented a policy that does not exist.

Hand a naive pointwise judge these two with a rubric like 'rate helpfulness 1 to 5,' and it will very plausibly score B higher. B is friendlier, more actionable, more on-brand. The judge is rewarding tone and completeness -- which is exactly what verbosity bias and helpfulness bias look like in the wild. You ship B. You have now automated the promising of refunds you don't give.

The fix is not a smarter judge. It's a differently-shaped question: reference-based, grounded, and run in both orderings.

import json
from anthropic import Anthropic

client = Anthropic()

RUBRIC = (
    "You are grading a customer-support answer for FACTUAL GROUNDING first, "
    "tone second. An answer that is warm but states a policy NOT supported by "
    "the reference is WORSE than a dry answer that is correct. "
    "Reward only claims entailed by the reference. "
    "Penalize hard: any invented policy, number, or guarantee."
)

def judge_pair(question, reference, answer_first, answer_second):
    prompt = (
        RUBRIC + "\n\n"
        "QUESTION:\n" + question + "\n\n"
        "REFERENCE (ground truth policy):\n" + reference + "\n\n"
        "ANSWER 1:\n" + answer_first + "\n\n"
        "ANSWER 2:\n" + answer_second + "\n\n"
        "First reason step by step about which claims are supported by the "
        "reference. THEN decide. Output JSON only: "
        "{\"reasoning\": str, \"winner\": \"1\" or \"2\" or \"tie\"}."
    )
    resp = client.messages.create(
        model="claude-opus-4-20250514",
        max_tokens=800,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(resp.content[0].text)

def judge_debiased(question, reference, ans_a, ans_b):
    # Run BOTH orderings to cancel position bias.
    fwd = judge_pair(question, reference, ans_a, ans_b)
    rev = judge_pair(question, reference, ans_b, ans_a)
    # In fwd, '1' means A wins. In rev, '2' means A wins.
    a_wins = (fwd["winner"] == "1") + (rev["winner"] == "2")
    b_wins = (fwd["winner"] == "2") + (rev["winner"] == "1")
    if a_wins > b_wins:
        return "A"
    if b_wins > a_wins:
        return "B"
    return "tie_or_position_dependent"

Three changes did the work, and none of them is 'use a bigger judge. ' First, the rubric names the failure mode explicitly and inverts the default preference -- a correct-but-dry answer must beat a warm-but-invented one. Second, it's reference-based, so the judge checks claims against the actual policy instead of vibing. Third, it runs both orderings and only trusts a verdict that survives the swap -- if flipping the order flips the winner, that's the judge telling you it graded position, not substance.

That last point is the sleeper feature. The 'tie_or_position_dependent' return value is not a failure -- it's a diagnostic. A high rate of position-dependent verdicts across your eval set means the judge isn't actually discriminating quality on those items, and any aggregate score built on them is noise wearing a lab coat.

Which shape to use, and when

The single biggest reliability lever is not the judge model -- it's the question shape. Here's how I actually choose:

• Pointwise (score 1-5): use for coarse triage and regression alarms, where you need an absolute number per item and cheap throughput. Do NOT lean on it when a 0.2-point difference is supposed to mean something -- absolute scores drift between runs and cluster around 3-4, so small gaps are usually noise. Cost: one call per item.
• Pairwise (A vs B): use when comparing two systems or two prompts and you want a trustworthy winner. It's the most stable shape because relative judgments are easier than absolute ones. The catch: comparing every candidate to every other is quadratic. For a leaderboard, anchor everything against one fixed baseline instead. Cost: two calls per pair, since you run both orderings.
• Reference-based: use whenever you have gold answers or, better, a retrievable ground-truth document. This is the only shape that reliably catches hallucinated facts, because the judge checks entailment instead of guessing. If you have references and aren't using them, you're leaving your best signal on the table.
• Reasoning-before-score (in every shape): non-negotiable. Forcing the justification before the verdict measurably improves agreement with humans and gives you an audit trail. When a score looks wrong, you read the reasoning and see whether the judge misread the answer or you misworded the rubric.

And the honest part nobody puts in the tutorial: sometimes the right shape is no judge at all. An LLM judge is the wrong tool when --

• There IS a right answer. Grading code that either passes tests or doesn't, math with a checkable result, or extraction against a known schema? Run the tests, check equality, validate the schema. A judge here just adds cost, latency, and a fresh way to be wrong.
• The stakes are high and irreversible per item. Medical, legal, financial, safety. Use the judge to triage and route, never as the final signature. The judge is for scale; consequential single decisions still want a human.
• You'd be grading the judge's own family. A model favors outputs that look like its own -- self-preference bias. If your system-under-test and your judge are the same model, you're measuring family resemblance, not quality. Use a different vendor, or at least a different model.
• The quality you care about is invisible in text. Latency, tone-match to a brand voice you never wrote into the rubric, or 'does this actually resolve the ticket end to end. ' If you can't state it as a criterion, the judge can't grade it.

The production gotchas that bite six weeks later

Everything above is setup-time. These are the ones that get you after you've been running happily for a while.

Judge drift on model upgrades. Your judge is 'the latest strong model. ' The vendor silently updates it. Your scores shift half a point and you burn a day hunting a regression in your own system that was actually a change in the ruler. Pin the judge model version, and when you must upgrade it, re-run a frozen calibration set through both and record the offset -- the way you'd re-zero a scale.

Rubric overfitting. You tweak the rubric until the judge agrees with you on your 30 hand-graded examples. Congratulations, you've fit the rubric to 30 points. Hold out a fresh human-graded set the rubric was never tuned against, and check agreement there. If it drops, you overfit.

The tie rate is a health metric, not a nuisance. If pairwise ties spike, either your two systems really are indistinguishable (fine, and useful to know) or your judge has stopped discriminating on this slice (not fine). Track it. A sudden tie-rate jump is often the first sign your eval set has gone stale relative to what the model now does.

Calibrate once, in public, and keep the receipt. Hand-grade a sample -- even 40 or 50 items -- compute agreement between you and the judge, and write that number down next to your dashboard. Now every automated score has honest error bars. 'The judge agrees with humans about 85% of the time on this task' -- an illustrative figure, yours will differ -- is a completely different, and far more useful, statement than a naked 4.6.

The bottom line

LLM-as-a-judge turns ungradeable free-form output into numbers you can track across thousands of examples, and that is a superpower. But the number is a measurement from a biased, correlated instrument, and the bias points straight at your model's most flattering failures. Name the failure mode in the rubric, ground it in a reference, run both orderings, judge with a different family, and calibrate against humans you actually trust. Do that and the judge stops being a mirror that tells you you're the fairest of them all -- and becomes the one instrument leveraged enough to let you ship models fast without shipping regressions blind.