Question 7
Suppose successive gradients keep pointing in a similar direction over several updates. Compared with plain gradient descent using the same nominal learning rate, momentum tends to:
Cancel updates in that direction
Build velocity in that direction while smoothing short-term gradient fluctuations
Make the gradient exactly unbiased
Force every parameter to use the same accumulated gradient