Machine learning foundations

Overview

Machine learning is the practice of fitting a program’s behaviour to data instead of writing the behaviour by hand. You choose a family of functions with adjustable parameters, a loss that scores how badly a given setting of those parameters does on examples, and a procedure that pushes the loss down. Then you measure the result on data the procedure never saw. That recipe is the same for a straight line through twenty points and for a model with hundreds of billions of parameters, which is why this hub comes first.

There are three broad settings. Supervised learning has a label for every example and learns to predict it: a price, a class, the next token. Unsupervised learning has no labels and looks for structure: clusters, low-dimensional projections, density. Self-supervised learning sits in between and builds the label out of the data itself, for instance by hiding part of an input and predicting it, which is how language models and most vision backbones are pretrained. Reinforcement learning, where the signal is a delayed reward, has its own hub.

For a working engineer these ideas matter because they are the diagnostic vocabulary. When a model is fine in the notebook and poor in production, the explanation is nearly always one of: the training data did not match the deployment data, the loss was not the metric anyone cared about, the model had too much or too little capacity, or the evaluation was leaky. None of those are library questions.

Key ideas

Learning is minimising a loss over examples

A model is a function with parameters. For a linear model the parameters are a weight vector and a bias, for a neural network they are millions of matrix entries, but either way the question is the same: which parameter values make the predictions agree with the examples. A loss function turns “agree” into a number. Squared error suits regression. Cross-entropy suits classification, because it treats the model output as a probability and penalises confident wrong answers hard while giving a smooth gradient everywhere. Training minimises the average loss over the training set, which statisticians call empirical risk minimisation.

The loss is a proxy. You rarely care about cross-entropy itself, you care about precision at a threshold, revenue, or whether a user accepts the suggestion. Keep the proxy and the real metric apart in your reports, and check that improving one improves the other.

Bias, variance and generalisation

A model that cannot represent the pattern underfits: training and validation error are both high. That is bias. A model flexible enough to fit noise in the training set overfits: training error is low and validation error is high. That is variance. For squared error the expected test error decomposes exactly into squared bias, variance and irreducible noise, and the classical picture is a U-shaped validation curve as capacity grows.

The classical picture is incomplete for modern networks. Zhang et al. showed in 2016 that standard networks can memorise random labels, and Belkin et al. showed in 2018 that validation error can fall again past the point of interpolation, the double descent curve. The practical lesson survives both results: plot training and validation error against capacity and against data size, and read the gap. Remedies for variance are more data, regularisation, early stopping and ensembling. The remedy for bias is a richer model or better features, which is where data and features comes in.

Optimisation: gradient descent and its relatives

Ordinary least squares has a closed-form solution. Almost nothing else does, so training means iteration: compute the gradient of the loss with respect to the parameters, step a small distance against it, repeat. The step size, the learning rate, is the first hyperparameter to sweep and the one most often wrong. Stochastic gradient descent estimates the gradient on a minibatch rather than the whole set, which is cheaper and adds noise that helps escape poor regions. Momentum smooths the steps. Adam scales each parameter’s step by a running estimate of its gradient variance and is the default in deep learning. For convex losses such as logistic regression any of these finds the global minimum. For neural networks they find a local one that generalises about as well as any other.

Evaluation is part of the method, not an afterthought

Split the data before looking at it. The validation set is for choosing between models and settings. The test set is for one final number, and every time you peek at it and change something it becomes a validation set. With small data, k-fold cross-validation reuses every example for both roles in turn at k times the cost. Pick the metric to match the decision the model supports: accuracy is meaningless on a 1 percent positive rate, and a ranked list needs a ranking metric. The evaluation hub and the metrics reference cover the choices.

Where to start

Read the beginner lessons in order: What is machine learning? for the vocabulary, Data is the model for splits and leakage, Linear and logistic regression from scratch to see parameters and a loss written out, The training loop for gradient descent, then Evaluation that does not lie. After that, Decision trees and gradient boosting shows that the same ideas hold for models with no gradients at all.

Tools and platforms

Common mistakes

• Reporting a test-set number after tuning on the test set. The number is a validation score with a misleading name.
• Watching training loss fall and calling it learning. Only the validation curve tells you whether the model generalises.
• Using accuracy on imbalanced classes. A model that always predicts the majority class scores 99 percent on a 1 percent event.
• Skipping the baseline. Predicting the mean or the majority class, or last week’s value, is the number every model must beat.
• Running one seed. Two runs that differ by 0.5 percent may be the same model with different noise.

Going deeper

• Bishop, Pattern Recognition and Machine Learning, 2006, free PDF from the publisher’s research page.
• Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, 2nd edition 2009, free PDF.
• Zhang et al., Understanding deep learning requires rethinking generalization, 2016.
• Belkin et al., Reconciling modern machine learning practice and the bias-variance trade-off, 2018.
• Kingma and Ba, Adam: A Method for Stochastic Optimization, 2014.
• The scikit-learn model evaluation guide and cross-validation guide.