Evaluation and benchmarks

Overview

Evaluation is the part of machine learning that decides whether anything else worked. A model is a claim about future data, and evaluation is the experiment that tests the claim. Done well, it tells you which of two models to ship and how confident to be. Done badly, it tells you what you wanted to hear, and the error only surfaces in production.

This hub is for engineers who have to produce a number and defend it. It covers the classical toolkit (metrics, splits, confidence intervals), the problems that arrived with large pretrained models (benchmark contamination, saturated leaderboards, evaluation by another model) and the online methods (human rating, A/B tests) that remain the only ground truth for products. The metric definitions live in /reference/evaluation-metrics/, and the beginner lesson /learn/beginner/06-evaluation-that-does-not-lie/ walks through the basics with code.

Key ideas

The held-out set is a one-shot instrument

A test set answers one question honestly: how does this model do on data it has never influenced. Every time you look at the test score and change something, the set leaks a little information into your decisions, and after enough iterations the test score is a validation score with a better name. The discipline is to split three ways (train, validation, test), tune only on validation, and read the test set rarely. Time-series and grouped data need splits that respect time and group boundaries, otherwise near-duplicates straddle the split and the score is inflated. /learn/beginner/02-data-is-the-model/ covers leakage in detail.

Report uncertainty with the number. A bootstrap over the test set gives a confidence interval in a few lines, and a difference between two models that lies inside the interval is not a result.

Benchmarks saturate and leak

Public benchmarks are held-out sets shared by everyone, and they are on the internet, which means they are in pretraining corpora. Contamination can be direct (the test questions appear verbatim in training data) or indirect (paraphrases, solutions, discussions). Zhang et al. (2024) rebuilt GSM8K with new problems of the same style and found several models scored noticeably lower on the fresh set than on the original. Detection methods include n-gram overlap against the training corpus (only possible when you own the corpus), canary strings planted in the benchmark, and comparing scores on original and perturbed versions. A benchmark score read from a leaderboard needs the benchmark name, the date and the evaluation harness, because all three change the number.

Benchmarks also saturate. MMLU (Hendrycks et al., 2020) was a strong discriminator in 2021 and by 2024 the frontier models were clustered near its label-noise ceiling. The field responds by writing harder sets, which then go through the same cycle. Treat any public benchmark as a screening test, not a decision.

Judges, raters and users disagree in predictable ways

Using a strong model to grade another model’s output scales well and correlates reasonably with human preference on chat tasks, as Zheng et al. (2023) showed with MT-Bench. It also has known biases: judges prefer the first of two answers, prefer longer answers, prefer their own family’s style, and can be fooled by confident wrong text. The mitigations are mechanical. Swap the order and average, use a rubric with named criteria rather than a single score, control for length, and measure the judge against a human-rated sample before trusting it on the rest.

Human evaluation is the reference but not the truth. Two raters given the same rubric agree less than you expect, so measure agreement (Cohen’s kappa or Krippendorff’s alpha), pilot the rubric, and pay for enough ratings that the interval is narrower than the difference you care about. A/B tests measure what users do rather than what raters say, which is better and different. They need a primary metric chosen before the test, guardrail metrics for things that must not get worse, a power calculation that says how long to run, and resistance to peeking.

Where to start

Read /learn/beginner/02-data-is-the-model/ for splits and leakage, then /learn/beginner/06-evaluation-that-does-not-lie/ for metrics, baselines and confidence intervals with code. /learn/beginner/09-using-an-llm-well/ shows how to build a small evaluation set for a prompt before changing it. For language model work, run one task from a public harness end to end and read the prompt template it used, because the template is part of the result. The /projects/tiny-rag-local-embeddings/ project includes a retrieval evaluation you can extend.

Tools and platforms

Common mistakes

• Reporting accuracy on an imbalanced set. A 99 percent score on a 1 percent positive class is the trivial baseline. Use precision, recall, AUROC or the metric your cost structure implies.
• Tuning on the test set, one glance at a time, then calling the final number held-out.
• Quoting a leaderboard score without the benchmark name, date and harness.
• Letting the judge model grade its own family without a human-calibrated sample.
• Running an A/B test until the result looks significant, then stopping. Decide the duration first.
• Building an evaluation set from easy, typical inputs. Include the failures from production and the cases you argued about.

Going deeper

• Hendrycks et al., Measuring Massive Multitask Language Understanding, 2020.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023.
• Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic, 2024.
• Liang et al., Holistic Evaluation of Language Models, 2022.
• Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020, the standard A/B testing textbook.
• scikit-learn model evaluation documentation.