Confusion matrices and evaluation metrics

Learning objectives

• Read a confusion matrix.
• Calculate accuracy, precision, recall, and F1 score.
• Explain why one metric is not enough.
• Connect metrics to real-world consequences.

Confusion matrix

For a binary classifier:

A false negative is a missed positive case. A false positive is an incorrect alert. The meaning depends on the problem.

Calculate metrics

from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)

actual = [1, 1, 1, 0, 0, 0]
predicted = [1, 0, 1, 1, 0, 0]

print(confusion_matrix(actual, predicted))
print(accuracy_score(actual, predicted))
print(precision_score(actual, predicted))
print(recall_score(actual, predicted))
print(f1_score(actual, predicted))

Accuracy is the fraction of all predictions that are correct. Precision asks, “When the model predicts positive, how often is it right?” Recall asks, “Of the actual positives, how many did it find?” F1 combines precision and recall into one harmonic-mean score.

Choose the metric first

• Use recall when missing a positive case is especially costly.
• Use precision when false alarms are especially costly.
• Use accuracy only when classes and error costs are reasonably balanced.
• Use F1 when both precision and recall matter.

Practice

Create two prediction lists with the same accuracy but different false-positive and false-negative counts. Decide which is safer for:

1. an obstacle detector;
2. a school email spam filter; and
3. a content recommendation system.

Error analysis

Metrics summarize behavior; they do not explain individual mistakes. Inspect representative false positives and false negatives, then ask whether the data, label, feature, threshold, or model caused the error.

Checkpoint

Can a model have high accuracy and poor recall? Construct a heavily imbalanced example and explain why.

Capstone handoff

Use these lessons to complete a model comparison project with a baseline, two models, a metric table, error analysis, and model card. The project rubric defines the expected evidence.