How to evaluate an LLM system without fooling yourself

You cannot improve what you do not measure, and with language models it is unusually easy to measure the wrong thing.

Start with a golden set, not a metric#

A hundred real questions with human-verified answers beats ten thousand synthetic ones. Draw them from actual traffic, weight them toward the hard cases, and freeze them.

LLM-as-judge needs its own evaluation#

A judge model is a model. It has biases: it prefers longer answers, it prefers its own family's outputs, and it is inconsistent across runs.

Before trusting a judge, label 200 examples by hand and measure agreement. Below Cohen's κ of 0.7, the judge is adding noise, not signal.

Gate on regressions in CI#

- name: Evaluation suite
  run: |
    python -m evals.run --suite core --baseline main
    python -m evals.gate --max-regression 0.02

A 2% tolerance is deliberate: below that, run-to-run variance dominates, and a zero-tolerance gate will block every pull request.

The metrics that survived contact with production#

• Faithfulness — is every claim supported by retrieved context?
• Answer relevance — does it address what was asked?
• Recall@k — was the evidence retrievable at all?
• Refusal correctness — does it decline when it should?

That last one is the most neglected and the most damaging when it is wrong.