Deep Learning Quiz 2: 1 December 2024 (September 2024 term)

Given a quadratic loss function $L(w) = w^2$, where $w$ represents the model parameter, you are using the AdaGrad optimizer with Stochastic Gradient Descent to minimize this loss. The learning rate $\eta$ is set to 1, and at the initial iteration ($t=0$), the parameter $w$ has a starting value of $w=2$.

AdaGrad Update Rule:

$v_t = v_{t-1} + (\nabla w_t)^2$

$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t$

$v_{-1} = 0$

$\text{use } \epsilon = 0$

Based on the above data, answer the given subquestions.

Given a quadratic loss function $L(w) = w^2$, where $w$ represents the model parameter, you are using the AdaGrad optimizer with Stochastic Gradient Descent to minimize this loss. The learning rate $\eta$ is set to 1, and at the initial iteration ($t=0$), the parameter $w$ has a starting value of $w=2$.

AdaGrad Update Rule:

$v_t = v_{t-1} + (\nabla w_t)^2$

$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t$

$v_{-1} = 0$

$\text{use } \epsilon = 0$

Based on the above data, answer the given subquestions.

Given a quadratic loss function $L(w) = w^2$, where $w$ represents the model parameter, you are using the AdaGrad optimizer with Stochastic Gradient Descent to minimize this loss. The learning rate $\eta$ is set to 1, and at the initial iteration ($t=0$), the parameter $w$ has a starting value of $w=2$.

AdaGrad Update Rule:

$v_t = v_{t-1} + (\nabla w_t)^2$

$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t$

$v_{-1} = 0$

$\text{use } \epsilon = 0$

Based on the above data, answer the given subquestions.