Optimization and gradients

Learning objectives

• describe a loss function as a measure of model error;
• explain a gradient as a direction and rate of change;
• simulate gradient descent on a simple function; and
• connect learning rate choices to stability and convergence.

Loss and improvement

A model has parameters, such as weights. A loss function compares a prediction with the desired result. Optimization changes the parameters to reduce loss on the training data.

For one parameter, consider:

def loss(weight):
    return (weight - 3) ** 2

weight = 0.0
learning_rate = 0.1

for step in range(10):
    gradient = 2 * (weight - 3)
    weight -= learning_rate * gradient
    print(step, round(weight, 3), round(loss(weight), 3))

The minimum of this function is at weight = 3. The loop uses the slope to move toward the minimum. A learning rate that is too small makes progress slow; one that is too large can overshoot or diverge.

Training vocabulary

• Parameter: a value learned by the model.
• Loss: error measured for one example or a batch.
• Gradient: how loss changes when a parameter changes.
• Learning rate: the step size used during an update.
• Epoch: one pass through the training data.
• Batch: the subset used for one update.
• Validation loss: error on held-out development data, used to monitor generalization.

Practice

Change the learning rate to 0.01, 0.5, and 1.1. Record whether the loss decreases smoothly, slowly, or unstably. Predict the behavior before running each version.

Common mistakes

• Assuming lower training loss always means better real-world performance.
• Changing several hyperparameters without recording them.
• Evaluating on the test set during every training decision.
• Treating a gradient as a guarantee that the next step improves every example.

Checkpoint

Why should a validation set be separate from the final test set? Explain how overfitting can appear in training and validation curves.

References

• Deep learning
• PyTorch optimization tutorial