Data, features and pipelines

Overview

A model is a compressed summary of its training data. Change the data and you change the model, usually more than any change to the architecture would. This hub covers the work that happens before and around training: deciding what to collect, getting labels, cleaning, building features, splitting so that evaluation means something, and keeping track of which data produced which model. It is for engineers who have discovered that the modelling step was the easy part.

The reason this matters is asymmetric. A better optimiser moves a benchmark by a fraction of a point. A duplicated row that straddles the train and test split moves it by ten points in the wrong direction, and nothing in the training logs will tell you. Kapoor and Narayanan surveyed 294 papers across 17 fields in 2022 and found leakage in every field, often enough to reverse the headline claim. The same mistakes appear in company code and are cheaper to prevent than to find.

Key ideas

The split comes first

Split the data before you clean it, before you look at distributions, before you fit a scaler. Anything you learn from the whole dataset, including the mean used for imputation or the vocabulary of a tokeniser, is information that leaks from the test set into the model. The scikit-learn Pipeline and ColumnTransformer exist so that every preprocessing step is fitted on the training fold only and then applied to the others. Use them even when the preprocessing seems harmless.

How you split depends on how the model will be used. A random split is right only when future examples are exchangeable with past ones. Time-series data needs a split by date, with the test period after the training period. Data with groups, such as several images per patient or several sessions per user, needs a split by group, or the model learns to recognise the patient instead of the disease. Near-duplicates need deduplication across the split, a real concern for web-scraped text and for images with multiple crops.

Leakage has several shapes

Target leakage is a feature that is a consequence of the label rather than a cause: a “claim paid” column in a fraud dataset, a timestamp of the cancellation in a churn dataset. Temporal leakage is any feature computed with information from after the prediction time, which is common when features are joined from a warehouse that has been updated since. Group leakage is the patient problem above. Preprocessing leakage is fitting a scaler, an encoder or a feature selector on the whole dataset. Duplicate leakage is the same or nearly the same example on both sides of the split.

The test for all of them is the same: for each feature, ask what the system would know at the moment it has to predict, and compute the feature only from that. A suspiciously good validation score is a leakage alarm, not good news.

Labels are a product, not a given

Most supervised projects live or die on labelling. Write annotation guidelines with worked edge cases before anyone labels anything, have two or more people label an overlapping sample, and measure agreement with Cohen’s kappa or Krippendorff’s alpha. Disagreement between annotators is an upper bound on what the model can reach and a lower bound on what the guidelines need. Northcutt et al. estimated in 2021 that around 3.3 percent of labels in common benchmark test sets are wrong, including ImageNet, so expect noise and budget for a relabelling pass on the examples the model gets most confidently wrong.

Labelling with a large language model is fine as a first pass if a human audits a random sample and the audit result is reported alongside the model’s score. See prompting and agents for how to set that up.

Features, schemas and versions

For tabular data, feature engineering is still where the gains are: ratios, lags, counts over windows, target encoding done inside cross-validation. Trees cope with raw numeric columns, while linear models and neural networks want scaled inputs, see classical ML. For text, images and audio the feature extractor is now usually a pretrained model, and the engineering shifts to choosing it and to embeddings.

Whatever the features, write down the schema and validate it on every batch: column types, allowed ranges, null rates, category sets. pandera and Great Expectations do this. Version the data with the same seriousness as the code: DVC stores content hashes so that a model can be traced to the exact bytes it was trained on. A datasheet, in the sense of Gebru et al., answers the questions a future colleague will ask about why the data was collected and what it does not cover.

Where to start

Start with Data is the model, which builds a leaky pipeline and then fixes it. Then Linear and logistic regression from scratch to see why scaling matters, Evaluation that does not lie for grouped and temporal splits, and Decision trees and gradient boosting for what trees do and do not need from features. The tabular classifier project puts a Pipeline behind an API so that the same preprocessing runs in production.

Tools and platforms

Common mistakes

• Fitting a scaler, imputer or encoder on the full dataset and then splitting. Every learned transform belongs inside the pipeline.
• Splitting randomly when examples share a user, a patient, a device or a day. Split by the group the model will meet fresh in production.
• Keeping a feature that is computed after the prediction time because it was in the warehouse export.
• Treating the label as truth. Measure annotator agreement and look at the examples where the model and the label disagree most confidently.
• Training on a dataset nobody can reproduce because it was a CSV on a laptop. Hash it, version it, and record the hash with the model.

Going deeper

• Kapoor and Narayanan, Leakage and the Reproducibility Crisis in ML-based Science, 2022.
• Gebru et al., Datasheets for Datasets, 2018.
• Northcutt, Athalye and Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, 2021.
• Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015.
• The scikit-learn guide to common pitfalls and recommended practices.
• The DVC documentation on versioning data and pipelines.