Your LLM-as-Judge is Lying to You

Here's a scenario I see constantly: a team ships an LLM-powered feature, sets up GPT-4 as a "judge" to score outputs, sees a beautiful 85% pass rate on their dashboard, and ships to production with confidence.

Three weeks later, users are complaining. The product feels off. But the evals say everything is fine.

What happened?

The judge was lying. Not maliciously—but systematically, in ways that are now well-documented and surprisingly consistent across every major model.

The Dirty Secret of LLM-as-Judge

I've been obsessed with evaluation systems since my early days working with LLMs. The promise of LLM-as-Judge is seductive: why pay for expensive human evaluators when you can have GPT-4 grade outputs at scale? Why wait days for feedback when you can get scores in seconds?

The problem is that these judges carry biases that can completely invalidate your evaluation pipeline. And most teams don't even know to look for them.

Recent research has identified at least 12 distinct bias types that undermine LLM judges. But three of them are so pervasive that every team using automated evaluation needs to understand them:

1. Position Bias: Where You Put It Changes the Score

Put two responses in a pairwise comparison. The judge will systematically favor one position—sometimes the first (primacy bias), sometimes the last (recency bias). This isn't a small effect.

Studies using the CALM framework found that swapping the order of responses can flip the verdict entirely. The same response that "wins" when shown first might "lose" when shown second.

Think about what this means for your A/B tests. If you're comparing your new model against your baseline and consistently putting the baseline first, your evaluation is compromised. Every. Single. Time.

The fix: Run each comparison twice with swapped positions. Only count a win if the same response wins in both positions. Yes, this doubles your eval cost. No, you can't skip it.

2. Verbosity Bias: More Words = Higher Scores

LLM judges love long responses. Verbose, meandering answers that take 500 words to say what could be said in 50? The judge thinks they're great.

This is particularly insidious because verbosity often correlates with worse user experience, not better. Users want concise answers. Your judge wants essays.

I've seen teams accidentally train their models to be more verbose because that's what the judge rewards. They're optimizing for a metric that makes the product worse. Classic Goodhart's Law in action.

The fix: Include explicit instructions in your judge prompt about valuing conciseness. Better yet, add a separate length-aware evaluation that penalizes unnecessary verbosity. Or even simpler: actually read your outputs. (More on that later.)

3. Self-Preference Bias: Models Love Themselves

This one should make you uncomfortable. Research published in late 2024 found that GPT-4 exhibits a 10% higher win rate for its own outputs when acting as a judge. Claude is even worse—showing a 25% self-preference bias in some evaluations.

Let that sink in. If you're using GPT-4 to evaluate GPT-4 outputs, the judge has a measurable preference for the defendant.

The mechanism is fascinating: it's not that models literally recognize their own outputs. Instead, they prefer responses with lower perplexity—text that is more "familiar" to their internal model of language. And what's more familiar than outputs that follow the same patterns they'd generate themselves?

The fix: Never use the same model family for both generation and evaluation. If you're generating with GPT-4, evaluate with Claude. If you're generating with Claude, evaluate with Gemini. Cross-pollinate your judges.

The Meta-Eval Problem

Here's where it gets philosophically tricky: how do you evaluate your evaluator?

If you can't trust your judge to give unbiased scores, and you don't have ground truth labels for everything, how do you know if your eval pipeline is even measuring reality?

This is the meta-eval problem, and it's one of the hardest challenges in production AI systems.

There's no silver bullet, but here's what I've seen work:

Calibration samples: Maintain a small set of examples with ground-truth human labels. Check if your judge agrees with humans on these examples. If agreement drops, your judge is drifting.
Multi-judge voting: Run the same evaluation through 3+ different judges (different model families). If they disagree, that's a signal—either the case is genuinely ambiguous, or your judges have conflicting biases.
Inversion tests: Deliberately corrupt some outputs in obvious ways. Does your judge catch them? If it can't identify obviously bad outputs, its "high scores" are meaningless.
Regular audits: Here's the uncomfortable truth—you still need humans in the loop. Not for every evaluation, but for regular audits of judge behavior. At minimum, manually review 20-50 outputs weekly.

Look at Your Data

This is the advice that Hamel Husain drums into everyone who takes his evals course, and he's right: look at your actual data.

Not dashboards. Not aggregated metrics. The actual, raw, individual outputs your system is producing.

I've watched teams spend weeks building elaborate evaluation infrastructure—multiple judges, weighted scoring, regression detection—only to realize they'd never actually read the outputs they were evaluating. When they finally did, the problems were obvious. No judge needed.

The judge-first approach feels scientific. It feels scalable. But it's often a way of avoiding the uncomfortable work of understanding your system's failure modes through direct observation.

Start with error analysis. Hamel's advice: spend 30 minutes reviewing 20-50 LLM outputs whenever you make significant changes. Write evaluators for errors you discover, not errors you imagine.

The 70% Rule

Be wary of optimizing for high eval pass rates.

If your system is passing 100% of your evals, you're not challenging it enough. You've built a system that can pass the tests you wrote, which may have nothing to do with whether it can handle real-world inputs.

A 70% pass rate often indicates a more meaningful evaluation. It means you're actually stress-testing your application, finding the edges where it breaks.

High scores feel good. They make the dashboard green. But they can be misleading, especially when your judge has all the biases we've discussed.

What Actually Works

After working with dozens of AI products, here's my current thinking on LLM-as-Judge systems that actually work:

Layer Your Evaluations

Don't rely on a single judge. Use:

Level 1: Rule-based checks – Fast, deterministic, catches obvious failures
Level 2: LLM judge – For nuanced quality, with bias mitigations
Level 3: Human audit – Sample-based, calibrates your entire pipeline

Keep Your Judge Simple

I've seen evaluation prompts that are 2,000 tokens long, with complex rubrics and multi-dimensional scoring. These almost always fail.

Simple judges outperform complex ones. A single yes/no question ("Does this response fully answer the user's question?") is often more reliable than a 5-dimension rubric with weighted scoring.

Why? Because complex rubrics create more opportunities for the judge's biases to manifest. Each dimension is another place where verbosity bias or position bias can creep in.

Accept the Cost

Doing this right costs money. You'll run evaluations multiple times with swapped positions. You'll use multiple model families as judges. You'll pay humans to audit samples.

This is the cost of building AI systems that actually work in production. The alternative—shipping with a biased eval system you don't understand—is more expensive in the long run.

The Bottom Line

LLM-as-Judge systems are powerful, but they're not objective. They carry systematic biases that can undermine your entire evaluation pipeline.

Position bias makes order matter when it shouldn't. Verbosity bias rewards length over quality. Self-preference bias means your judge is not impartial.

The solution isn't to abandon automated evaluation—it's to approach it with appropriate skepticism:

Use multiple judges from different model families
Swap positions and average results
Maintain calibration samples with human labels
Actually look at your data regularly
Don't celebrate high pass rates; interrogate them

Your LLM-as-Judge might be lying to you. But now you know how to catch it.

Building AI systems that need reliable evaluation? I help companies set up eval pipelines that actually measure reality. Let's talk.