Data is the model: datasets, splits and leakage

Lesson 01 ended with the claim that the only score worth having is the one on examples a model has never seen. This lesson is about how to get that score and how it goes wrong. You will build a small customer dataset in pandas, cut it into three pieces, train a model on one piece and score it on the others. Then you will add a single column that quietly encodes the answer, watch the held-out score jump from 0.78 to 0.97, and see that score collapse the moment the model meets the real world. If you take one habit from the beginner track, make it the suspicion this lesson teaches.

Why the data matters more than the algorithm

A model can only learn what is in its examples. If the examples are wrong, biased, stale or contaminated, the most sophisticated algorithm will faithfully learn the wrongness. If the examples are clean and plentiful, a linear model from 1805 is often enough. Experienced practitioners spend most of their time on data and a small fraction on model choice, and beginners do the reverse. The title of this lesson is a reminder of where the time should go.

The data also sets the ceiling. In the dataset below, the label is generated with deliberate randomness, so even the perfect model cannot score 100 %. Knowing that ceiling exists, and that nobody tells you where it is on a real dataset, is the start of reading scores sensibly.

A dataset you can trust because you made it

Real datasets hide their generating rule. A synthetic one lets you write the rule down, which means you know exactly what a model ought to discover and can tell when it has discovered something else. The dataset here is a small customer churn table: four features about each customer and a label saying whether they left.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 1000

age = rng.integers(18, 70, size=n)
tenure_months = rng.integers(1, 120, size=n)
monthly_spend = np.round(rng.gamma(shape=2.0, scale=25.0, size=n), 2)
support_calls = rng.poisson(lam=1.2, size=n)

# The hidden rule that generates the label. The model has to rediscover it.
logit = (0.5
         - 0.03 * tenure_months
         + 1.0 * support_calls
         - 0.02 * monthly_spend
         + 0.01 * (age - 40))
p_churn = 1.0 / (1.0 + np.exp(-logit))
churned = (rng.random(n) < p_churn).astype(int)

df = pd.DataFrame({
    "age": age,
    "tenure_months": tenure_months,
    "monthly_spend": monthly_spend,
    "support_calls": support_calls,
    "churned": churned,
})
print(df.head())
print("rows:", len(df), " churn rate:", round(df["churned"].mean(), 3))

   age  tenure_months  monthly_spend  support_calls  churned
0   62             66          44.43              0        1
1   51             10         160.95              2        0
2   44             23          10.79              3        1
3   32            105          38.69              1        0
4   34             53          14.29              0        0
rows: 1000  churn rate: 0.346

Read the rule. Each extra support call adds 1.0 to a running score, each month of tenure subtracts 0.03, each pound of monthly spend subtracts 0.02, and age nudges it slightly. The score is pushed through a squashing function so it becomes a probability between 0 and 1, and a coin weighted by that probability decides the label. For a customer with 3 support calls, 23 months of tenure and spend of about 11, the score is 0.5 minus 0.69 plus 3.0 minus 0.22 plus 0.04, about 2.6, which is a churn probability of roughly 0.93. Row 2 above is that customer, and the coin came up churned. Row 0, with no calls and 66 months of tenure, had a probability near 0.25 and churned anyway. That is the randomness, and no model can predict it.

The squashing function and the weighted score are exactly the logistic regression model of lesson 03. The dataset is built so that lesson 03’s model is the right one for it.

Three splits and what each is for

Before anything is fitted, the rows are divided into three groups.

One dataset cut into three. Training rows fit the parameters. Validation rows guide every decision you make about the model. Test rows are scored once, at the end, and never used to choose anything.

The training set is what the model fits. Its parameters are adjusted until the loss on these rows is low, so a score on these rows is flattering and tells you little.

The validation set is what you use to make decisions. Should the model be a linear one or a tree. Should it use four features or five. Should it train for ten rounds or a hundred. Each choice is made by comparing validation scores, and over a long project you will look at this set hundreds of times. That is the problem: every decision made against it leaks a little information from it into the model, and after enough decisions the validation score is flattering too.

The test set exists to solve that. It is scored once, at the very end, and its number is the one you report. If you look at it, make a change and look again, it has become a second validation set and you no longer have an honest number.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(0)
n = 1000
age = rng.integers(18, 70, size=n)
tenure_months = rng.integers(1, 120, size=n)
monthly_spend = np.round(rng.gamma(shape=2.0, scale=25.0, size=n), 2)
support_calls = rng.poisson(lam=1.2, size=n)
logit = (0.5 - 0.03 * tenure_months + 1.0 * support_calls
         - 0.02 * monthly_spend + 0.01 * (age - 40))
churned = (rng.random(n) < 1.0 / (1.0 + np.exp(-logit))).astype(int)
df = pd.DataFrame({"age": age, "tenure_months": tenure_months,
                   "monthly_spend": monthly_spend,
                   "support_calls": support_calls, "churned": churned})

features = ["age", "tenure_months", "monthly_spend", "support_calls"]
X = df[features]
y = df["churned"]

# First cut: 80 % for fitting, 20 % held out as the test set.
X_rest, X_test, y_rest, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0, stratify=y)
# Second cut: of the 80 %, keep a quarter (20 % overall) for validation.
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)

print("train:", len(X_train), " val:", len(X_val), " test:", len(X_test))
print("churn rate per split:",
      round(y_train.mean(), 3), round(y_val.mean(), 3), round(y_test.mean(), 3))

model = LogisticRegression(max_iter=2000)
model.fit(X_train, y_train)
print("train accuracy:", round(accuracy_score(y_train, model.predict(X_train)), 3))
print("val accuracy:  ", round(accuracy_score(y_val, model.predict(X_val)), 3))
print("majority-class baseline on val:", round(max(y_val.mean(), 1 - y_val.mean()), 3))

train: 600  val: 200  test: 200
churn rate per split: 0.347 0.345 0.345
train accuracy: 0.755
val accuracy:   0.775
majority-class baseline on val: 0.655

Three details in the code are habits worth copying. random_state=0 makes the split reproducible, so a colleague running the same code gets the same rows in each group. stratify=y keeps the churn rate the same in every split, which matters when one class is rare: a random 200-row cut of a dataset with 5 % churn could easily contain 4 % or 7 %, and the score would swing with it. And the baseline line at the end is the score you would get by predicting “did not churn” for everyone. A model that cannot beat the baseline has learned nothing, however good its accuracy sounds. Here 0.775 against 0.655 is a real improvement, and the gap between train and validation is small, so the model is not memorising.

The validation score is a little above the training score, which looks odd until you remember that both are estimates from a few hundred rows. Differences of a point or two are noise. Lesson 06 puts error bars on these numbers and introduces cross-validation, which reuses the rows more carefully.

Target leakage: a feature that knows the answer

Now the demonstration. Suppose the billing system has a column called final_bill_issued. When a customer leaves, the system issues a final bill and sets the flag. A few customers who stay get the flag too, through billing errors. It looks like an ordinary feature, it is in the table next to the others, and a model will love it.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(0)
n = 1000
age = rng.integers(18, 70, size=n)
tenure_months = rng.integers(1, 120, size=n)
monthly_spend = np.round(rng.gamma(shape=2.0, scale=25.0, size=n), 2)
support_calls = rng.poisson(lam=1.2, size=n)
logit = (0.5 - 0.03 * tenure_months + 1.0 * support_calls
         - 0.02 * monthly_spend + 0.01 * (age - 40))
churned = (rng.random(n) < 1.0 / (1.0 + np.exp(-logit))).astype(int)
df = pd.DataFrame({"age": age, "tenure_months": tenure_months,
                   "monthly_spend": monthly_spend,
                   "support_calls": support_calls, "churned": churned})

# The leak: a column filled in by the billing system AFTER a customer leaves.
# Churned customers get a final-bill flag 95 % of the time, others 3 %.
final_bill_issued = np.where(
    df["churned"] == 1,
    (rng.random(n) < 0.95).astype(int),
    (rng.random(n) < 0.03).astype(int),
)
df["final_bill_issued"] = final_bill_issued

clean = ["age", "tenure_months", "monthly_spend", "support_calls"]
leaky = clean + ["final_bill_issued"]
y = df["churned"]

def fit_and_score(cols):
    X_rest, X_test, y_rest, y_test = train_test_split(
        df[cols], y, test_size=0.2, random_state=0, stratify=y)
    X_train, X_val, y_train, y_val = train_test_split(
        X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)
    m = LogisticRegression(max_iter=2000).fit(X_train, y_train)
    return (accuracy_score(y_val, m.predict(X_val)),
            accuracy_score(y_test, m.predict(X_test)))

for name, cols in [("clean", clean), ("leaky", leaky)]:
    val_acc, test_acc = fit_and_score(cols)
    print(f"{name:5s}  val {val_acc:.3f}  test {test_acc:.3f}")

# What the leaky model would score at the moment you actually need it:
# before the customer has left, when final_bill_issued is always 0.
X_rest, X_test, y_rest, y_test = train_test_split(
    df[leaky], y, test_size=0.2, random_state=0, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)
m = LogisticRegression(max_iter=2000).fit(X_train, y_train)
X_deploy = X_test.copy()
X_deploy["final_bill_issued"] = 0
print("leaky model in deployment:", round(accuracy_score(y_test, m.predict(X_deploy)), 3))
print("majority baseline:        ", round(max(y_test.mean(), 1 - y_test.mean()), 3))

clean  val 0.775  test 0.765
leaky  val 0.965  test 0.965
leaky model in deployment: 0.665
majority baseline:         0.655

Look at the second line. With the leaky column, validation accuracy is 0.965 and so is test accuracy. Every honest check passed. The split was random, stratified and reproducible, the test set was touched once, and the number is excellent. A reader of this report would ship the model.

The last two lines are what happens next. The purpose of a churn model is to predict who will leave before they do, so that someone can intervene. At that moment, no final bill has been issued for anyone, so the column is zero for every customer the model is asked about. Scored on that reality, the leaky model gets 0.665, one point above the baseline of guessing “stays” for everybody. It is worse than the clean model that scored 0.765, and the 0.965 it reported was a lie.

The splits did not catch it because they cannot. Splitting protects you from a model that memorises rows. It does nothing about a column that carries the label into both halves of the split. The flag was set after the label was known, so it is a consequence of the label dressed up as a feature. That is target leakage, and the only defence is to ask of every column: at the moment the prediction will be made, will this value exist, and will it have been computed without knowing the answer.

Kind	How it happens	What catches it
Target leakage	A feature is derived from, or written after, the label	Ask when each column is filled in
Preprocessing leakage	A scaler, imputer or encoder is fitted on all rows before splitting, so test statistics shape the training data	Fit every transform on training rows only, and use a pipeline that enforces it
Temporal leakage	Rows are shuffled, so the model trains on next month and is tested on last month	Split by time, with training strictly before validation and test
Duplicate leakage	The same record, or a near copy, lands in both training and test	Deduplicate before splitting, and compare the two sets for exact matches
Group leakage	Several rows belong to one entity (patient, user, device) and are spread across splits, so the model recognises the entity rather than the pattern	Split by group, keeping all of an entity’s rows together

What a suspicious score looks like

A few patterns should make you stop and check the data before celebrating.

The score is far better than the baseline and far better than the problem seems to deserve. Predicting churn from four weak signals at 0.965 is not plausible. Predicting it at 0.775 is.

One feature dominates. If you fit a tree (lesson 05) and its first split explains almost everything, look hard at that column.

Train, validation and test agree too well. Honest models usually show a small gap between training score and the others. A model that is near-perfect everywhere has either an easy problem or a leak.

The score is high but the model has no plausible mechanism. A model predicting hospital readmission with 99 % accuracy from “number of follow-up appointments booked” is reading the discharge paperwork, not predicting the future.

None of these is proof. All of them are reasons to find the column, trace where it comes from and, if in doubt, drop it and see how far the score falls. The fall is the size of the leak.

Exercise

Change the leak so that final_bill_issued is set for only 60 % of churned customers instead of 95 %. Re-run the leaky block. How much does the validation score fall, and is it still enough to fool a reader.
Add a customer_id column that is 0 for the first 500 rows and 1 for the rest, and generate the second half with a higher base churn rate. Split with train_test_split as above and then split again by taking the first 800 rows for training and the last 200 for test. Compare the two test scores and explain the difference using the table in the previous section.
Fit a StandardScaler on all 1,000 rows, transform, then split and train. Then fit the scaler on the training rows only. Report both validation scores. The difference will be tiny here. Write one sentence on why it is still worth doing properly.

Summary

A model learns only what is in its examples, so time spent on the data pays off more than time spent on the algorithm.
Training rows fit the parameters, validation rows guide every decision, and test rows are scored once at the end. Rows never appear in more than one group.
Always print a baseline. A score that does not beat predicting the majority class means nothing.
Target leakage is a feature that carries the answer. A random split cannot catch it, so ask of every column when it is written and by what.
Leaks also arrive through preprocessing fitted on all rows, shuffled time series, duplicates and groups split across sets. Each has a specific check.

Data is the model: datasets, splits and leakage

You will learn