The Limits of Public Benchmarks
Public benchmarks are useful for tracking the field and poor for choosing a model for a specific job.
Three problems. Contamination — benchmark questions leak into training data, inflating scores without reflecting capability. Optimisation pressure — labs know which benchmarks matter commercially and optimise toward them. Relevance — a score on graduate-level physics reasoning says little about how a model will handle your customer support tickets.
Read benchmarks as a rough capability tier, not as a ranking to select from.
Building Your Own Eval Set
The single highest-value thing most teams can do is assemble 50 to 200 real examples from their actual use case, with known-good outputs.
Sources: real user queries from logs, edge cases that previously failed, examples covering the range of inputs you expect, and deliberately difficult cases.
This set becomes the thing you run against every model, every prompt change and every version upgrade. Without it, model selection and prompt iteration are guesswork dressed as judgement.
Ways to Score Outputs
Exact match — where there is one correct answer. Cheap, objective, limited.
Human review — most accurate and least scalable. Use on a sample.
LLM-as-judge — a model scores outputs against a rubric. Scales well, correlates reasonably with human judgement when the rubric is specific, and carries known biases including preferring longer answers and favouring outputs from the same model family.
Assertion checks — programmatic rules on the output: is it valid JSON, does it cite a source, does it stay under length, does it avoid forbidden content. Underused and very effective.
What to Measure Beyond Accuracy
Latency, because a better answer that arrives too late is not better. Cost per request at realistic volume. Consistency — run the same input several times and see how much the answer varies. Failure mode — when the model is wrong, is it obviously wrong or plausibly wrong? Plausible wrongness is far more dangerous.
Running Evals Continuously
Model providers update models. A prompt tuned six months ago may behave differently today, silently.
Run the eval set on a schedule and on every change to prompts, models or retrieval. Treat a drop in eval score the way you would treat a failing test, because that is what it is.
Building an Eval Set That Is Worth Trusting
A private eval set on your own task is worth more than every public benchmark combined, and building one is less work than it sounds.
Start with fifty real examples. Drawn from actual usage, not invented, and including the awkward ones — the ambiguous request, the badly-typed input, the case with no good answer. A set of clean examples measures a situation you do not have.
Include known failures deliberately. Every time something goes wrong in production, that example joins the set. This is what makes an eval set improve over time rather than decay.
Write the expected outcome, not the expected wording. For most tasks there are many acceptable answers, so define what must be true of a good response rather than fixing its text.
Keep it private and keep it out of prompts. An eval set that leaks into training data or into a system prompt stops measuring anything.
And version it. When the set changes, the scores are not comparable to last month's — which is the most common way a team convinces itself a model got better.
What to Measure Besides Whether It Was Right
Accuracy is one dimension and rarely the one that determines whether a feature survives contact with users.
Consistency. Run the same input several times. A model that is right on average and varies wildly is unusable in a workflow where someone expects the same answer twice.
Failure mode. When it is wrong, is it obviously wrong or plausibly wrong? A system that fails visibly is safer than one with a better score that fails convincingly — the same argument that governs the choice between model sizes.
Refusal behaviour at both ends: refusing things it should do is as damaging as attempting things it should not.
Latency distribution, not the average. The slow tail is what users experience as broken.
Cost per successful outcome, which is the only cost figure that compares options fairly — a cheaper model needing two attempts is not cheaper.
And drift. Re-run the set on a schedule against a hosted model, because it changes without notifying you, and the first sign is usually a user complaint rather than a metric.