Deep Learning End Term: 13 April 2025 (January 2025 term)

A neural network is being trained for a regression problem in which the target is in the interval (−1, 1). Which of the following is NOT a good choice for the activation function at the output layer?

In the transformer architecture, which of the following are true regarding cross attention in the decoder?

Consider applying dropout to a hidden layer with p = 0.5. Which of the following are true?

Consider the following plot of error versus model complexity. Models of varying complexity are evaluated on the same training and test datasets. Model complexity increases from left to right:

Select all true options.

Consider the following tasks which can be solved using an encoder-decoder architecture. Which of these tasks has a decoder where an RNN is not necessary?

The input volume to a convolutional layer is 100×100×5. If ten kernels, each of size 9 × 9, with unit stride are applied over this volume, what should be the padding so that the output volume has dimensions 100 × 100 × 10? Note that you should enter the value of P as per the convention we have been following.

Based on the above data, answer the given subquestions.

Find the value of x at which the tower attains its maximum value.

Based on the above data, answer the given subquestions.

Find the maximum value that the tower attains.

Consider the multi-head self-attention mechanism in the encoder of a transformer with 8 heads. The word embedding dimension is 32. In a given head, the query, key and value vectors have the same dimension and each of them is 4. The sequence length is 5.
Based on the above data, answer the given subquestions.

In any given head, what is the dimension of the WQ matrix?

Consider the multi-head self-attention mechanism in the encoder of a transformer with 8 heads. The word embedding dimension is 32. In a given head, the query, key and value vectors have the same dimension and each of them is 4. The sequence length is 5.
Based on the above data, answer the given subquestions.

In any given head, what is the dimension of the query-key product matrix?

Consider the multi-head self-attention mechanism in the encoder of a transformer with 8 heads. The word embedding dimension is 32. In a given head, the query, key and value vectors have the same dimension and each of them is 4. The sequence length is 5.
Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Find the value y at the end of the forward pass.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Which of the following is true?

Based on the above data, answer the given subquestions.

Find a.

Based on the above data, answer the given subquestions.

Find b.

Based on the above data, answer the given subquestions.

Find c.

Consider the CBOW model for learning word embeddings with embedding dimension two. The window size is one, that is, we only use the previous word as the context to predict the current word. The vocabulary is made up of the words {“one”, “two”, “three”, “four”}. At some point during training, the word and context matrices are given below. The first column in each matrix corresponds to the embedding for “one”, the second column is for “two” and so on:

$W_{\text{word}} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ -1 & 1 & 2 & -2 \end{bmatrix}, \quad W_{\text{context}} = \begin{bmatrix} 0 & 1 & 0 & 1 \\ -1 & 0 & 1 & 1 \end{bmatrix}$

The sample that has come up now during training is “two three”. Note that “three” is to be considered as the true label here. Enter your answer correct to two places after the decimal for both sub-questions.

Based on the above data, answer the given subquestions.

Find the probability of predicting “three” given “two” as context.

Consider the CBOW model for learning word embeddings with embedding dimension two. The window size is one, that is, we only use the previous word as the context to predict the current word. The vocabulary is made up of the words {“one”, “two”, “three”, “four”}. At some point during training, the word and context matrices are given below. The first column in each matrix corresponds to the embedding for “one”, the second column is for “two” and so on:

$W_{\text{word}} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ -1 & 1 & 2 & -2 \end{bmatrix}, \quad W_{\text{context}} = \begin{bmatrix} 0 & 1 & 0 & 1 \\ -1 & 0 & 1 & 1 \end{bmatrix}$

The sample that has come up now during training is “two three”. Note that “three” is to be considered as the true label here. Enter your answer correct to two places after the decimal for both sub-questions.

Based on the above data, answer the given subquestions.

For a multi-class classification problem with five classes, consider the following CNN architecture:

• The first column is the type of layer.
• The second column is the layer specification. If it is an FC layer, it is the number of neurons. If it is a convolution or pooling layer, it is the information pertaining to kernels. NA refers to “Not Applicable” wherever this information is not needed. In a convolutional/pooling layer, $K$ is the number of filters, $F$ is the spatial dimension of the filter, $P$ is the padding and $S$ is the stride.
• The third column corresponds to the activation volumes output by the non-FC layers. For example, the input layer passes on a volume of size $64 \times 64 \times 3$ to “Convolution-1”, which outputs a volume of size $V_1$. Each volume is of the type $W \times H \times D$.

Note that $V_5$ is flattened before it is passed on to “FC-1”. Output layer is also considered as an FC layer with a softmax activation function.

Based on the above data, answer the given subquestions.

Find the number of parameters associated with the layer “Convolution-2” that are required to transform V2 to V3. Ignore biases.

For a multi-class classification problem with five classes, consider the following CNN architecture:

• The first column is the type of layer.
• The second column is the layer specification. If it is an FC layer, it is the number of neurons. If it is a convolution or pooling layer, it is the information pertaining to kernels. NA refers to “Not Applicable” wherever this information is not needed. In a convolutional/pooling layer, $K$ is the number of filters, $F$ is the spatial dimension of the filter, $P$ is the padding and $S$ is the stride.
• The third column corresponds to the activation volumes output by the non-FC layers. For example, the input layer passes on a volume of size $64 \times 64 \times 3$ to “Convolution-1”, which outputs a volume of size $V_1$. Each volume is of the type $W \times H \times D$.

Note that $V_5$ is flattened before it is passed on to “FC-1”. Output layer is also considered as an FC layer with a softmax activation function.

Based on the above data, answer the given subquestions.

Which of the following corresponds to V4, the activation volume output by the layer “MaxPooling- 2”?

For a multi-class classification problem with five classes, consider the following CNN architecture:

• The first column is the type of layer.
• The second column is the layer specification. If it is an FC layer, it is the number of neurons. If it is a convolution or pooling layer, it is the information pertaining to kernels. NA refers to “Not Applicable” wherever this information is not needed. In a convolutional/pooling layer, $K$ is the number of filters, $F$ is the spatial dimension of the filter, $P$ is the padding and $S$ is the stride.
• The third column corresponds to the activation volumes output by the non-FC layers. For example, the input layer passes on a volume of size $64 \times 64 \times 3$ to “Convolution-1”, which outputs a volume of size $V_1$. Each volume is of the type $W \times H \times D$.

Note that $V_5$ is flattened before it is passed on to “FC-1”. Output layer is also considered as an FC layer with a softmax activation function.

Based on the above data, answer the given subquestions.

The parameters associated with the fully connected layers, namely “FC- 1” and “Output”, represent what percentage of the total number of parameters in the network? The answer should be rounded off to the nearest integer. Ignore biases.

Based on the above data, answer the given subquestions.

With a total of 7 time steps (T = 7), what is the total count of parameters (including bias) within the network?

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

If all the parameters (including bias) in the network are initialized to zero, what will be the total loss after 10 time steps (assume that indices start with 1) for the input $[0,0,1,0,0,0]^T$? The ground truth $(y \in \mathbb{R}^5)$ for each time step is given by the following sequence $[y_0, y_4, y_1, y_2, y_3, y_4, y_4, y_4, y_2, y_3]$. Assume the loss to be cross-entropy at each time step. (Use natural log and write the answer correct up to two decimal places.) Each $y$ is one-hot vector (i.e $y_0$ means $y = [1,0,0,0,0]^T$, $y_4$ means $y = [0,0,0,0,1]^T$ and so on).

Based on the above data, answer the given subquestions.

Which of these is the correct ordering among the learning rates?

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.