Deep Learning Quiz 2: 16 March 2025 (January 2025 term)

What is the impact of using a sigmoid activation function on the gradients when the input to the neuron is a very large negative number?

Why is bias correction needed in the Adam optimizer?

How do skip connections in ResNet work?

What is the primary purpose of L2 regularization in machine learning models?

What is the soft target used when injecting noise at the output?

Consider a neural network architecture for a regression task. Three distinct subsets of the training dataset are used to train three distinct models: A,B and C. Note that these models have the same architecture with different parameter values. Consider a test dataset of size four with true labels: y = [1,3,6,8]. The predictions of the models on this dataset is given below:
• Model A predictions: [1, 4, 7, 8]
• Model B predictions: [3, 5, 6, 9]
• Model C predictions: [2, 3, 5, 7]
Calculate the variance of these models averaged over all data-points in the test dataset. Enter your answer up to 2 decimal places.

The weighted moving average (WMA) is used to smooth data by giving more importance to recent values. Suppose we compute a weighted moving average for a sequence of gradient values using the formula:

$v_t = \beta v_{t-1} + (1 - \beta)\nabla w_t$

where,

• $v_t$ is the weighted moving average at time step $t$.
• $\beta = 0.9$ is the decay factor.
• $\nabla w_t$ is the gradient value at time step $t$.
• $v_{-1} = 0$

If the gradient values at $t = 0, 1, 2$ are $\nabla w_0 = 5, \nabla w_1 = 3$ and $\nabla w_2 = 4$. Compute $v_2$. Enter your answer correct up to 3 decimal places.

Given an input array $X$ and a kernel/filter $K$ as follows:

$X = \begin{bmatrix} 1 & 0 & -2 & 1 \\ 1 & 2 & 0 & 1 \\ -1 & -1 & 1 & 0 \\ 0 & 1 & 0 & 1 \end{bmatrix}$

$K = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 0 & 1 \\ 1 & 0 & 1 \end{bmatrix}$

Based on the above data, answer the given subquestions.

Given an input array $X$ and a kernel/filter $K$ as follows:

$X = \begin{bmatrix} 1 & 0 & -2 & 1 \\ 1 & 2 & 0 & 1 \\ -1 & -1 & 1 & 0 \\ 0 & 1 & 0 & 1 \end{bmatrix}$

$K = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 0 & 1 \\ 1 & 0 & 1 \end{bmatrix}$

Based on the above data, answer the given subquestions.

Given an input array $X$ and a kernel/filter $K$ as follows:

$X = \begin{bmatrix} 1 & 0 & -2 & 1 \\ 1 & 2 & 0 & 1 \\ -1 & -1 & 1 & 0 \\ 0 & 1 & 0 & 1 \end{bmatrix}$

$K = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 0 & 1 \\ 1 & 0 & 1 \end{bmatrix}$

Based on the above data, answer the given subquestions.

Pass $B$ through the sigmoid (logistic) function with weight equal to 1 and no bias to get the final output $\hat{y}$.

$L$ is some loss function that depends on $\hat{y}$. Given that $\frac{\partial L}{\partial \hat{y}} = -1$, determine the value of $\frac{\partial L}{\partial K_{00}}$ where $K_{00}$ is the element of $K$ at index $(0,0)$. Note that we use zero-indexing here.

Enter your answer correct up to 3 digits after the decimal.

Consider a loss function, $L(w) = w^3 + 2w$, where $w$ represents the model parameter. The AdaGrad optimizer is used to minimize this loss. The learning rate $\eta$ is set to $0.5$, and at the initial iteration ($t = 0$), the parameter $w$ has a starting value of $w_0 = 2$.

AdaGrad Update Rule:

$\begin{aligned} v_t &= v_{t-1} + (\nabla w_t)^2 \\ w_{t+1} &= w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t \\ v_{-1} &= 0 \\ \text{use } \epsilon &= 0 \end{aligned}$

Enter your answer correct up to two decimal places for all sub-questions. For intermediate calculations, use three decimal places so that your final answer falls within the expected range.

Based on the above data, answer the given subquestions.

Consider a loss function, $L(w) = w^3 + 2w$, where $w$ represents the model parameter. The AdaGrad optimizer is used to minimize this loss. The learning rate $\eta$ is set to $0.5$, and at the initial iteration ($t = 0$), the parameter $w$ has a starting value of $w_0 = 2$.

AdaGrad Update Rule:

$\begin{aligned} v_t &= v_{t-1} + (\nabla w_t)^2 \\ w_{t+1} &= w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t \\ v_{-1} &= 0 \\ \text{use } \epsilon &= 0 \end{aligned}$

Enter your answer correct up to two decimal places for all sub-questions. For intermediate calculations, use three decimal places so that your final answer falls within the expected range.

Based on the above data, answer the given subquestions.

Consider a loss function, $L(w) = w^3 + 2w$, where $w$ represents the model parameter. The AdaGrad optimizer is used to minimize this loss. The learning rate $\eta$ is set to $0.5$, and at the initial iteration ($t = 0$), the parameter $w$ has a starting value of $w_0 = 2$.

AdaGrad Update Rule:

$\begin{aligned} v_t &= v_{t-1} + (\nabla w_t)^2 \\ w_{t+1} &= w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t \\ v_{-1} &= 0 \\ \text{use } \epsilon &= 0 \end{aligned}$

Enter your answer correct up to two decimal places for all sub-questions. For intermediate calculations, use three decimal places so that your final answer falls within the expected range.

Based on the above data, answer the given subquestions.

Consider a loss function, $L(w) = w^3 + 2w$, where $w$ represents the model parameter. The AdaGrad optimizer is used to minimize this loss. The learning rate $\eta$ is set to $0.5$, and at the initial iteration ($t = 0$), the parameter $w$ has a starting value of $w_0 = 2$.

AdaGrad Update Rule:

$\begin{aligned} v_t &= v_{t-1} + (\nabla w_t)^2 \\ w_{t+1} &= w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t \\ v_{-1} &= 0 \\ \text{use } \epsilon &= 0 \end{aligned}$

Enter your answer correct up to two decimal places for all sub-questions. For intermediate calculations, use three decimal places so that your final answer falls within the expected range.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.