Deep Learning Quiz 2: 3 August 2025 (May 2025 term)

Consider four training experiments, each involving a neural network trained on the same dataset for 50 epochs using gradient descent. The loss curves for these models are shown below. The model obtained at the end of 50 epochs is retained in each of the four cases. The solid curve is the training loss and the dashed curve is the validation loss.

Now consider these four statements.
(A) Learning rate used is too high
(B) Low bias and high variance
(C) Low bias and low variance
(D) High bias and low variance
Choose the most appropriate pairing of models with the statements.

Which of the following optimizers use an adaptive learning rate?

An autoencoder is a feedforward neural network in which the input and the output layer have the same number of neurons. There is one hidden layer in between. The number of neurons in the hidden layer is 30% of the neurons present in the input layer. Call the two weight matrices $W_1$ and $W_2$. Ignore biases everywhere. We now consider two instances of this architecture:

• Network-1: $W_1$ and $W_2$ are not tied, meaning, they are two different weight matrices with no parameters being shared.
• Network-2: $W_1$ and $W_2$ are tied, meaning, $W_2 = W_1^T$ with parameters being shared.

Both networks are trained on some dataset and their parameters are saved on a hard-disk with efficient storage being the focus. If the number of parameters that need to be stored in networks 1 and 2 are $n_1$ and $n_2$ respectively, find $n_1/n_2$.

The input volume to a convolutional layer of a CNN has shape 64 × 64 × 8, where 8 is the depth or the number of channels. The conv-layer has 16 kernels each of spatial extent 5 × 5. The stride is 1 and padding is 2. Ignore biases.
Based on the above data, answer the given subquestions.

Which of these is the shape of the output volume after the convolution operation?

The input volume to a convolutional layer of a CNN has shape 64 × 64 × 8, where 8 is the depth or the number of channels. The conv-layer has 16 kernels each of spatial extent 5 × 5. The stride is 1 and padding is 2. Ignore biases.
Based on the above data, answer the given subquestions.

Find the total number of parameters associated with the convolutional layer.

The input volume to a convolutional layer of a CNN has shape 64 × 64 × 8, where 8 is the depth or the number of channels. The conv-layer has 16 kernels each of spatial extent 5 × 5. The stride is 1 and padding is 2. Ignore biases.
Based on the above data, answer the given subquestions.

If the input volume to the convolutional layer is 128 × 128 × 8, would the number of parameters associated with the conv layer change?

Consider an ML model for which the un-regularized loss on some dataset $D$ as a function of parameter vector $w = (w_1, w_2)$ is $L_D(w) = w_1^2 + w_2^2 - 2w_1 - 2w_2 + 2$. Let the L2- regularization term be $L_R(w) = \frac{\lambda}{2} \cdot ||w||^2$, where $\lambda > 0$ is the regularization rate. The L2-regularized total loss is $L = L_D + L_R$. Define:

$\begin{aligned} w_D^* &= \underset{w}{\arg\min}\ L_D \\ w^* &= \underset{w}{\arg\min}\ L \end{aligned}$

arg min returns the value of $w$ at which the function attains its minimum.

The subquestions are independent of each other.

Based on the above data, answer the given subquestions.

Consider an ML model for which the un-regularized loss on some dataset $D$ as a function of parameter vector $w = (w_1, w_2)$ is $L_D(w) = w_1^2 + w_2^2 - 2w_1 - 2w_2 + 2$. Let the L2- regularization term be $L_R(w) = \frac{\lambda}{2} \cdot ||w||^2$, where $\lambda > 0$ is the regularization rate. The L2-regularized total loss is $L = L_D + L_R$. Define:

$\begin{aligned} w_D^* &= \underset{w}{\arg\min}\ L_D \\ w^* &= \underset{w}{\arg\min}\ L \end{aligned}$

arg min returns the value of $w$ at which the function attains its minimum.

The subquestions are independent of each other.

Based on the above data, answer the given subquestions.

Consider an ML model for which the un-regularized loss on some dataset $D$ as a function of parameter vector $w = (w_1, w_2)$ is $L_D(w) = w_1^2 + w_2^2 - 2w_1 - 2w_2 + 2$. Let the L2- regularization term be $L_R(w) = \frac{\lambda}{2} \cdot ||w||^2$, where $\lambda > 0$ is the regularization rate. The L2-regularized total loss is $L = L_D + L_R$. Define:

$\begin{aligned} w_D^* &= \underset{w}{\arg\min}\ L_D \\ w^* &= \underset{w}{\arg\min}\ L \end{aligned}$

arg min returns the value of $w$ at which the function attains its minimum.

The subquestions are independent of each other.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

For a regression problem, consider an image X of size 4×4 and true label y that is passed through a toy convolutional network that has one convolutional layer with a single kernel followed by a max- pooling layer. Ignore the bias.

W denotes the kernel. S is the stride. P is the padding and K is the number of kernels. F is the spatial extent of the kernel for the max-pooling operation.
Based on the above data, answer the given subquestions.

Which of the following is the output produced by the conv+relu layer?

For a regression problem, consider an image X of size 4×4 and true label y that is passed through a toy convolutional network that has one convolutional layer with a single kernel followed by a max- pooling layer. Ignore the bias.

W denotes the kernel. S is the stride. P is the padding and K is the number of kernels. F is the spatial extent of the kernel for the max-pooling operation.
Based on the above data, answer the given subquestions.

For a regression problem, consider an image X of size 4×4 and true label y that is passed through a toy convolutional network that has one convolutional layer with a single kernel followed by a max- pooling layer. Ignore the bias.

W denotes the kernel. S is the stride. P is the padding and K is the number of kernels. F is the spatial extent of the kernel for the max-pooling operation.
Based on the above data, answer the given subquestions.

If the loss is defined as $L(y, \hat{y}) = \frac{1}{2} \cdot (\hat{y} - y)^2$ and $W = \begin{bmatrix} W_{11} & W_{12} & W_{13} \\ W_{21} & W_{22} & W_{23} \\ W_{31} & W_{32} & W_{33} \end{bmatrix}$, then compute $\frac{\partial L}{\partial W_{13}}$.