Lesson 3 · Deep Learning
Training, Backpropagation & Gradient Descent
9 min
You'll be able to
- Explain loss as a measure of error
- Understand how backpropagation computes blame
- Describe gradient descent as a downhill walk
Training means adjusting weights to reduce error. A loss function scores how wrong the network is on the training data. Gradient descent then nudges each weight in the direction that lowers the loss — like walking downhill on an error surface.
Backpropagation is the algorithm that tells each weight how much it contributed to the error, using the chain rule of calculus. Combined, these two ideas train networks of any size.
learning_rate = 0.01
for step in range(1000):
prediction = forward(x, w) # pass data through
loss = (prediction - y) ** 2 # how wrong are we?
grad = backward(loss, w) # blame each weight
w = w - learning_rate * grad # nudge downhillChallenge
Walk the landscape
In your own words, explain what a 'local minimum' is on an error surface and why a tiny learning rate could get you stuck.
Knowledge Check
Training
What does gradient descent repeatedly do?
Backpropagation uses the chain rule to distribute error to each weight.
Answer all questions to submit.