ML baselines and train/test splits

Learning objectives

• Explain why a baseline is necessary.
• Separate training data from test data.
• Use a reproducible split.
• Recognize why evaluating on training data is misleading.

Why start with a baseline?

A baseline is a simple rule or prediction used for comparison. If a complex model does not beat the baseline on unseen data, complexity has not helped. For a regression problem, a baseline might always predict the training-set average. For classification, it might always predict the most common class.

Split the data

from sklearn.model_selection import train_test_split

features = [[1], [2], [3], [4], [5], [6]]
labels = [2, 4, 6, 8, 10, 12]

X_train, X_test, y_train, y_test = train_test_split(
    features,
    labels,
    test_size=0.33,
    random_state=42,
)

print(len(X_train), len(X_test))

random_state makes the example reproducible. The test set should be used for final evaluation, not repeatedly adjusted until the result looks good.

A classification baseline

from collections import Counter

y_train = ["cat", "cat", "dog", "cat"]
most_common = Counter(y_train).most_common(1)[0][0]
baseline_predictions = [most_common, most_common]

print(most_common)
print(baseline_predictions)

The baseline predicts the most common training label for every test example. A useful model must beat this rule for a meaningful metric.

Common mistakes

• Fitting preprocessing on the full dataset before splitting.
• Reporting training accuracy as final performance.
• Changing the test set after seeing results.
• Using a random split for time-ordered data without thinking about the future.

Practice

Create a baseline for a dataset with three classes. Calculate its accuracy on a small test list. Explain when accuracy may hide a poor result.

Checkpoint

Why is a test set called “unseen” data? What would happen if the model memorized every training example?

Next step

Continue to linear regression.