AI/ML Field Guide
Cheat sheet

Evaluation metrics

Classification, regression, ranking and generation metrics in four tables with the formula, when to use each and what it hides, plus runnable scikit-learn blocks and bootstrap confidence intervals.

Last verified 13 Nov 2026· scikit-learn 1.9, NumPy 2.5

This sheet is for anyone reporting a number and wanting it to mean something. Each table gives the formula, the situation it fits and the failure it hides. The blocks compute every metric on arrays small enough to check by hand. For the reasoning about splits and leakage see /learn/beginner/06-evaluation-that-does-not-lie/.

Classification

Notation: TP, FP, TN, FN are the four cells of the confusion matrix for the positive class. p is the predicted probability.

Metric Formula Use when What it hides
Accuracy (TP + TN) / N Classes balanced and errors cost the same With 1 % positives, predicting all negative scores 99 %
Precision TP / (TP + FP) A false alarm is expensive (spam, fraud review queues) Says nothing about the positives you missed
Recall (sensitivity) TP / (TP + FN) A miss is expensive (screening, safety) Predict everything positive and recall is 1
F1 2 PR / (P + R) One number for an imbalanced binary task Ignores true negatives, weights P and R equally whether or not that matches the cost
Matthews correlation (MCC) (TP TN - FP FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) Imbalanced data, you want all four cells to count Less familiar to readers, undefined when a row or column is empty
ROC AUC Probability a random positive scores above a random negative Comparing rankers independent of a threshold Optimistic on heavy imbalance, because FPR stays small when negatives are many
PR AUC (average precision) Area under precision against recall Heavy imbalance, the positive class is what you care about Depends on the base rate, so not comparable across datasets
Log loss (cross-entropy) -mean(y log p + (1 - y) log(1 - p)) You need calibrated probabilities One confident wrong prediction dominates
Calibration error (ECE) Mean over bins of abs(accuracy - confidence) Probabilities feed a downstream decision Bin choice changes the number

Multi-class: average="macro" treats every class equally (small classes count), "micro" pools the cells (large classes dominate), "weighted" is macro scaled by support. State which one you used.

import numpy as np
from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
                             roc_auc_score, average_precision_score, log_loss,
                             matthews_corrcoef, confusion_matrix)

y_true = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
y_prob = np.array([0.9, 0.8, 0.4, 0.3, 0.7, 0.2, 0.1, 0.05, 0.3, 0.15])
y_pred = (y_prob >= 0.5).astype(int)

print("confusion [[tn fp] [fn tp]]:", confusion_matrix(y_true, y_pred).tolist())
print(f"accuracy   {accuracy_score(y_true, y_pred):.3f}")
print(f"precision  {precision_score(y_true, y_pred):.3f}")
print(f"recall     {recall_score(y_true, y_pred):.3f}")
print(f"F1         {f1_score(y_true, y_pred):.3f}")
print(f"MCC        {matthews_corrcoef(y_true, y_pred):.3f}")
print(f"ROC AUC    {roc_auc_score(y_true, y_prob):.3f}")
print(f"PR AUC     {average_precision_score(y_true, y_prob):.3f}")
print(f"log loss   {log_loss(y_true, y_prob):.3f}")
confusion [[tn fp] [fn tp]]: [[5, 1], [2, 2]]
accuracy   0.700
precision  0.667
recall     0.500
F1         0.571
MCC        0.356
ROC AUC    0.896
PR AUC     0.854
log loss   0.455

Note the gap between the thresholded metrics (F1 0.571) and the ranking metrics (ROC AUC 0.896). The scores order the classes well and the 0.5 threshold is in the wrong place. Choose the threshold on a validation set for the cost you care about, never on the test set.

Regression

Metric Formula Use when What it hides
MAE mean(abs(y - yhat)) Errors cost linearly, you want the number in the units of y Treats a 10-unit miss as ten 1-unit misses
MSE mean((y - yhat)^2) Training loss, large errors matter more Units are squared, one outlier dominates
RMSE sqrt(MSE) Same as MSE but readable in units of y Still outlier-driven
MAPE mean(abs(y - yhat) / abs(y)) Scale-free comparison across series Explodes near y = 0, penalises over-prediction more than under
R squared 1 - SS_res / SS_tot Reporting how much variance is explained relative to predicting the mean Can be negative on a test set, rises with every added feature on the training set
Quantile (pinball) loss mean(max(q (y - yhat), (q - 1)(y - yhat))) Predicting intervals or percentiles Reported per quantile, one number per q 
import numpy as np
from sklearn.metrics import (mean_absolute_error, mean_squared_error, root_mean_squared_error,
                             r2_score, mean_absolute_percentage_error)

y_true = np.array([3.0, 5.0, 2.5, 7.0, 4.0, 10.0])
y_pred = np.array([2.8, 5.4, 2.0, 6.5, 4.2, 14.0])   # one large miss on the last point

print(f"MAE   {mean_absolute_error(y_true, y_pred):.3f}")
print(f"MSE   {mean_squared_error(y_true, y_pred):.3f}")
print(f"RMSE  {root_mean_squared_error(y_true, y_pred):.3f}")
print(f"MAPE  {mean_absolute_percentage_error(y_true, y_pred):.3f}")
print(f"R2    {r2_score(y_true, y_pred):.3f}")
print(f"R2 of predicting the mean: {r2_score(y_true, np.full_like(y_true, y_true.mean())):.3f}")
MAE   0.967
MSE   2.790
RMSE  1.670
MAPE  0.145
R2    0.580
R2 of predicting the mean: 0.000

Five of the six predictions are within 0.5. The single 4-unit miss makes RMSE nearly twice MAE. Report both and the reader can tell whether you have a few large errors or many small ones.

Ranking and retrieval

For one query with a ranked list and a set of relevant items. Average over queries to report.

Metric Formula Use when What it hides
Recall at k relevant in top k / all relevant RAG retrieval, where the generator only sees the top k Nothing about order within the k, and you need to know all the relevant items
Precision at k relevant in top k / k Fixed-size result pages Penalises queries with fewer than k relevant items
Mean reciprocal rank (MRR) mean over queries of 1 / rank of the first relevant item One right answer is enough (question answering, known-item search) Ignores every relevant item after the first
Mean average precision (MAP) mean over queries of the average of precision at each relevant position Several relevant items, order matters Binary relevance only
nDCG at k DCG / ideal DCG, DCG = sum of gain / log2(rank + 1) Graded relevance, order matters Needs graded labels, which are expensive
Hit rate at k 1 if any relevant item in top k Recommenders with one held-out item per user Coarse 
import numpy as np
from sklearn.metrics import ndcg_score

# One query. Ranked list of 6 document ids and which of them are relevant.
ranked = ["d3", "d7", "d1", "d9", "d2", "d5"]
relevant = {"d7", "d2", "d8"}          # d8 was never retrieved

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & relevant) / len(relevant)

def precision_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & relevant) / k

def reciprocal_rank(ranked, relevant):
    for i, d in enumerate(ranked, start=1):
        if d in relevant:
            return 1 / i
    return 0.0

def average_precision(ranked, relevant):
    hits, total = 0, 0.0
    for i, d in enumerate(ranked, start=1):
        if d in relevant:
            hits += 1
            total += hits / i
    return total / len(relevant)

for k in (1, 3, 5):
    print(f"recall@{k} {recall_at_k(ranked, relevant, k):.3f}  precision@{k} {precision_at_k(ranked, relevant, k):.3f}")
print(f"MRR {reciprocal_rank(ranked, relevant):.3f}  AP {average_precision(ranked, relevant):.3f}")

# nDCG uses graded relevance: scores are the ranker's, gains are the labels, in rank order
gains = np.array([[0, 2, 0, 0, 1, 0]])           # d7 highly relevant, d2 partly
scores = np.array([[6, 5, 4, 3, 2, 1]])           # descending, i.e. the ranked order
print(f"nDCG@3 {ndcg_score(gains, scores, k=3):.3f}  nDCG@6 {ndcg_score(gains, scores):.3f}")
recall@1 0.000  precision@1 0.000
recall@3 0.333  precision@3 0.333
recall@5 0.667  precision@5 0.400
MRR 0.500  AP 0.300
nDCG@3 0.480  nDCG@6 0.627

Recall at 5 is capped at 0.667 because d8 was never retrieved. That is the number to watch in a RAG system: if the gold passage is not in the top k, no prompt engineering downstream can recover it. See /reference/rag-pipeline-checklist/.

Generation

Metric Formula Use when What it hides
Perplexity exp(-mean log p(token)) Comparing language models on the same tokeniser and data Not comparable across tokenisers, says nothing about usefulness
BLEU Brevity penalty times geometric mean of n-gram precisions, n = 1 to 4 Machine translation with several references Zero when any n-gram order has no match, blind to meaning and word order beyond 4
ROUGE-L F-measure of the longest common subsequence Summarisation against a reference Rewards copying the source, punishes valid paraphrase
Exact match and token F1 String equality, or F1 over tokens Short-answer QA with a known answer Formatting differences count as wrong
BERTScore Cosine similarity of contextual embeddings, token-aligned Paraphrase-tolerant similarity to a reference Depends on the embedding model, scores cluster in a narrow band
LLM as judge A model scores the output against a rubric No reference exists, open-ended tasks Position and length bias, the judge’s own errors, and it must be validated against human labels
Pass at k Fraction of problems where any of k samples passes the tests Code generation Tests decide, and weak tests pass wrong code
Human preference (win rate) Fraction of pairwise comparisons won The final word on quality Expensive, slow, raters disagree 
import math
from collections import Counter

def ngrams(tokens, n):
    return Counter(tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1))

def bleu(candidate, reference, max_n=4):
    """Sentence-level BLEU with uniform weights, no smoothing. Zero if any n-gram order has no match."""
    cand, ref = candidate.split(), reference.split()
    log_precision = 0.0
    for n in range(1, max_n + 1):
        c, r = ngrams(cand, n), ngrams(ref, n)
        overlap = sum(min(count, r[g]) for g, count in c.items())
        total = max(sum(c.values()), 1)
        if overlap == 0:
            return 0.0
        log_precision += math.log(overlap / total) / max_n
    brevity = 1.0 if len(cand) > len(ref) else math.exp(1 - len(ref) / len(cand))
    return brevity * math.exp(log_precision)

def rouge_l(candidate, reference):
    """F-measure of the longest common subsequence of tokens."""
    a, b = candidate.split(), reference.split()
    table = [[0] * (len(b) + 1) for _ in range(len(a) + 1)]
    for i, x in enumerate(a, 1):
        for j, y in enumerate(b, 1):
            table[i][j] = table[i - 1][j - 1] + 1 if x == y else max(table[i - 1][j], table[i][j - 1])
    lcs = table[-1][-1]
    if lcs == 0:
        return 0.0
    p, r = lcs / len(a), lcs / len(b)
    return 2 * p * r / (p + r)

def perplexity(token_log_probs):
    return math.exp(-sum(token_log_probs) / len(token_log_probs))

ref = "the cat sat on the mat"
for cand in ["the cat sat on the mat", "the cat is on the mat", "a dog slept on a rug"]:
    print(f"{cand!r:32} BLEU {bleu(cand, ref):.3f}  ROUGE-L {rouge_l(cand, ref):.3f}")
print(f"perplexity of a 6-token sequence at log p = -2.0 each: {perplexity([-2.0] * 6):.2f}")
'the cat sat on the mat'         BLEU 1.000  ROUGE-L 1.000
'the cat is on the mat'          BLEU 0.000  ROUGE-L 0.833
'a dog slept on a rug'           BLEU 0.000  ROUGE-L 0.167
perplexity of a 6-token sequence at log p = -2.0 each: 7.39

The second candidate changes one word and scores zero on unsmoothed sentence BLEU because no 4-gram survives. Corpus-level BLEU and smoothed variants exist for this reason. Use a library (sacrebleu) for anything you publish, and name the variant.

Confidence intervals by bootstrap

A metric on a test set is a sample statistic. Resample the test items with replacement, recompute the metric a couple of thousand times, and read the 2.5th and 97.5th percentiles. Two models whose intervals overlap substantially are not distinguishable by that test set, whatever the leaderboard says. For a paired comparison, resample the same indices for both models and bootstrap the difference.

import numpy as np
from sklearn.metrics import f1_score

rng = np.random.default_rng(0)
n = 200
y_true = rng.integers(0, 2, n)
# a classifier that is right about 80 % of the time
y_pred = np.where(rng.random(n) < 0.8, y_true, 1 - y_true)

point = f1_score(y_true, y_pred)
boot = []
for _ in range(2000):
    idx = rng.integers(0, n, n)                 # resample test items with replacement
    boot.append(f1_score(y_true[idx], y_pred[idx]))
lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"F1 {point:.3f}  95 % bootstrap CI [{lo:.3f}, {hi:.3f}]  width {hi - lo:.3f}")

# halve the test set and the interval widens
half = n // 2
boot_half = [f1_score(y_true[idx], y_pred[idx]) for idx in (rng.integers(0, half, half) for _ in range(2000))]
lo2, hi2 = np.percentile(boot_half, [2.5, 97.5])
print(f"with n={half}: CI [{lo2:.3f}, {hi2:.3f}]  width {hi2 - lo2:.3f}")
F1 0.748  95 % bootstrap CI [0.680, 0.809]  width 0.129
with n=100: CI [0.632, 0.820]  width 0.188

With 200 test items the interval is 13 points wide. A reported improvement of 2 points on a test set that size is noise until shown otherwise. Report the interval, the test-set size and the number of seeds alongside every headline number.

Sources