Deep Learning End Term: 31 August 2025, Set QDB3 (May 2025 term)

Consider the following two statements regarding model performance:
Statement 1: A model achieving zero training loss is guaranteed to perform well on unseen data. Statement 2: Incorporating a regularization term in the loss function may lead to higher training loss but lower generalization error.
Which of the following options is correct?

How does unsupervised layerwise pretraining help in alleviating the vanishing gradient problem?

A dataset is given by

$X = \begin{bmatrix} 1 & 2 & 0 & 5 & 0 \\ 1 & 0 & 3 & 0 & 0 \\ 1 & 1 & 0 & 1 & 1 \\ 1 & -1 & 0 & 0 & 0 \end{bmatrix}, y = \begin{bmatrix} 10 \\ 6 \\ 5 \\ 2 \end{bmatrix}$

The rows of $X$ represent samples and the columns represent features, with the first column corresponds the bias term. We use a linear regression neuron where the prediction $\hat{y}_i$ for a sample $x_i$ is given by the linear combination $\hat{y}_i = z_i = \sum_{j=0}^{4} w_j x_{ij}$.

The weights are updated using Stochastic Gradient Descent (SGD) for one epoch (i.e., once for each of the 4 samples). The loss function is the Mean Squared Error, $L = (\hat{y} - y)^2$. If all weights are initialized to $w_j = 0.5$, which of the following weights is updated the fewest number of times?

Given the input matrix $X$ and kernel $K$:

$X = \begin{bmatrix} 2 & 1 & 0 & 1 \\ 0 & 1 & -1 & 0 \\ 3 & 2 & 1 & -1 \\ 1 & 0 & -2 & 2 \end{bmatrix}, \quad K = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$

• Perform convolution of $K$ over $X$ with stride $= 1$ and no padding to get matrix $A$.
• Apply average pooling on all of $A$ to produce scalar $B$.
• Apply the ReLU activation on $B$ to obtain final output $\hat{y}$.

If $\frac{\partial L}{\partial \hat{y}} = 2$, compute $\frac{\partial L}{\partial K_{11}}$, where $K_{11}$ is the centre element of the kernel. Submit the final answer correct to two decimal places.

Suppose you are given three encoder hidden states at time $t$:

$h_j = h_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \quad h_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}, \quad h_3 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$

The previous decoder hidden state is:

$s_{t-1} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$

Given the attention score function:

$\text{score}(s_{t-1}, h_j) = V_{att}^\top \tanh\left(U_{att} s_{t-1} + W_{att} h_j\right)$

where the hyperbolic tangent function is defined as:

$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = \frac{e^{2x} - 1}{e^{2x} + 1}$

where

$V_{att} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \quad U_{att} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}, \quad W_{att} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$

Based on the above data, answer the given subquestions.

Suppose you are given three encoder hidden states at time $t$:

$h_j = h_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \quad h_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}, \quad h_3 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$

The previous decoder hidden state is:

$s_{t-1} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$

Given the attention score function:

$\text{score}(s_{t-1}, h_j) = V_{att}^\top \tanh\left(U_{att} s_{t-1} + W_{att} h_j\right)$

where the hyperbolic tangent function is defined as:

$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = \frac{e^{2x} - 1}{e^{2x} + 1}$

where

$V_{att} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \quad U_{att} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}, \quad W_{att} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$

Based on the above data, answer the given subquestions.

Normalize the attention scores using the softmax function to obtain the attention weights $\alpha_{tj}$. submit $\alpha_{t1}$ (i.e first element of $\alpha$ vector). Submit the final answer correct to two decimal places.

$\alpha_{tj} = \text{align}(s_{t-1}, h_j) = \frac{\exp\left(\text{score}(s_{t-1}, h_j)\right)}{\sum_{i=1}^{n} \exp\left(\text{score}(s_{t-1}, h_i)\right)}$

Suppose you are given three encoder hidden states at time $t$:

$h_j = h_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \quad h_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}, \quad h_3 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$

The previous decoder hidden state is:

$s_{t-1} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$

Given the attention score function:

$\text{score}(s_{t-1}, h_j) = V_{att}^\top \tanh\left(U_{att} s_{t-1} + W_{att} h_j\right)$

where the hyperbolic tangent function is defined as:

$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} = \frac{e^{2x} - 1}{e^{2x} + 1}$

where

$V_{att} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \quad U_{att} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}, \quad W_{att} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$

Based on the above data, answer the given subquestions.

Calculate the context vector $c_t$

$c_t = \sum_{j=1}^{n} \alpha_{tj} h_j$

Provide the sum all the elements of $c_t$. Submit the final answer correct to two decimal places.

• Sequence Length : $t$
• Number of Heads : $h$
• Embedding dimension : $d_{\text{model}}$
• Input $X \in \mathbb{R}^{d_{\text{model}} \times t}$
• $d_k = d_q = \frac{d_{\text{model}}}{h}$
• $W_Q \in \mathbb{R}^{d_q \times d_{\text{model}}}$
• $W_K \in \mathbb{R}^{d_k \times d_{\text{model}}}$
• $W_V \in \mathbb{R}^{d_v \times d_{\text{model}}}$
• $W_o \in \mathbb{R}^{d_{\text{model}} \times (h \times d_v)}$

Based on the above data, answer the given subquestions.

Suppose $t = 32$, $d_{\text{model}} = 64$, $h = 2$ and $d_v = 16$. What will be the shape of the output of the scaled dot-product attention operation for a single head, given by:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Compute the resulting output dimension and report the total number of elements in the resulting attention output.

• Sequence Length : $t$
• Number of Heads : $h$
• Embedding dimension : $d_{\text{model}}$
• Input $X \in \mathbb{R}^{d_{\text{model}} \times t}$
• $d_k = d_q = \frac{d_{\text{model}}}{h}$
• $W_Q \in \mathbb{R}^{d_q \times d_{\text{model}}}$
• $W_K \in \mathbb{R}^{d_k \times d_{\text{model}}}$
• $W_V \in \mathbb{R}^{d_v \times d_{\text{model}}}$
• $W_o \in \mathbb{R}^{d_{\text{model}} \times (h \times d_v)}$

Based on the above data, answer the given subquestions.

• Sequence Length : $t$
• Number of Heads : $h$
• Embedding dimension : $d_{\text{model}}$
• Input $X \in \mathbb{R}^{d_{\text{model}} \times t}$
• $d_k = d_q = \frac{d_{\text{model}}}{h}$
• $W_Q \in \mathbb{R}^{d_q \times d_{\text{model}}}$
• $W_K \in \mathbb{R}^{d_k \times d_{\text{model}}}$
• $W_V \in \mathbb{R}^{d_v \times d_{\text{model}}}$
• $W_o \in \mathbb{R}^{d_{\text{model}} \times (h \times d_v)}$

Based on the above data, answer the given subquestions.

Assume a Feed-Forward Network (FFN) follows the Multi-Head Attention layer in the encoder. The FFN consists of two linear transformations:

$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$

$\text{where } W_1 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}, \quad W_2 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$

Calculate the total number of parameters in the FFN layer (including bias terms), where $d_{\text{ff}} = 256$ and $d_{\text{model}} = 64$.

Consider a CBOW model for learning word embeddings. The vocabulary is made up of three words, {good, bad, ugly}. W and C are the matrices that contain the word and context embeddings respectively. The columns in each matrix correspond to the embeddings. Both matrices are of shape 2 x 3:

The context window is 1, meaning, the next word is predicted using just the current word as context. Recall that we use softmax to make predictions at the output.
Based on the above data, answer the given subquestions.

Consider a CBOW model for learning word embeddings. The vocabulary is made up of three words, {good, bad, ugly}. W and C are the matrices that contain the word and context embeddings respectively. The columns in each matrix correspond to the embeddings. Both matrices are of shape 2 x 3:

The context window is 1, meaning, the next word is predicted using just the current word as context. Recall that we use softmax to make predictions at the output.
Based on the above data, answer the given subquestions.

The CBOW model is now used to generate a “sentence” or a string of words. First we pass the word “good” and retain the word with highest probability as the output, say word1, which is in turn passed as input to the model. If the model is run this way for exactly three time steps, what is the sentence that it outputs? Note that the sentence here is “word1 word2 word3”.

Consider a CBOW model for learning word embeddings. The vocabulary is made up of three words, {good, bad, ugly}. W and C are the matrices that contain the word and context embeddings respectively. The columns in each matrix correspond to the embeddings. Both matrices are of shape 2 x 3:

The context window is 1, meaning, the next word is predicted using just the current word as context. Recall that we use softmax to make predictions at the output.
Based on the above data, answer the given subquestions.

Now consider updating the word embeddings using the sample “good good”. The first “good” in the string is used as context and the second “good” as the true label. Use cross entropy as the loss function and run one iteration of gradient descent with η = 1 starting with the existing values for the embeddings.
Find the updated word embedding for “good” and choose the most appropriate option from below. Note that you have to compute the updated word embedding for “good” and not its context embedding.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.