Deep Learning End Term: 31 August 2025, Set QIC1 (May 2025 term)

Based on the two statements provided choose the correct option?
Statement 1: Optimization’s primary goal is to reduce the training error.
Statement 2: Regularization helps to prevent overfitting, which in turn reduces the training error.

Consider a feedforward neural network with one hidden layer trained using backpropagation for a binary classification task. The network has the following architecture:

• Input layer with 15 neurons
• Hidden layer with 25 neurons
• Output layer with 1 neuron

Assume that:

• The activation function for both the hidden and output layers is the sigmoid function, $\sigma(z)$,
• The derivative of the sigmoid function is $\sigma'(z) = \sigma(z)(1 - \sigma(z))$,
• The loss function used is binary cross-entropy (BCE),
• $\hat{y}^{(i)}$ is the output for the $i$-th example, $y^{(i)}$ is the true label, and $h_j^{(i)}$ is the output of the $j$-th hidden neuron for the $i$-th example,
• There are no biases in the network.

Let $w_j$ represent the weight connecting the $j$th neuron of the hidden layer to the output neuron. Identify the correct weight update rule $\Delta w_j$ for each type of gradient descent: Vanilla Gradient Descent (GD) ,mini-Batch Gradient Descent (BGD), and Stochastic Gradient Descent (SGD). (Recall: with BCE + sigmoid, the gradient w.r.t. the pre-activation simplifies to $\hat{y}^{(i)} - y^{(i)}$.

Scaled Dot-Product Attention

You are given a simple transformer attention setup with the following configuration:

• Embedding dimension: $d_{\text{model}} = 2$
• Input sequence (sequence length = 2, each column vector in $\mathbb{R}^2$):

$X = \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}$

• Dimension of queries and keys: $d_k = 2$
• Projection matrices:

$W_Q = W_K = \begin{bmatrix} 2 & -1 \\ 1 & -1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 2 \end{bmatrix}$

Scaled Dot-Product Attention:

$\text{Scaled Dot-Product Attention}(Q, K, V) = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Scaled Dot-Product Attention

You are given a simple transformer attention setup with the following configuration:

• Embedding dimension: $d_{\text{model}} = 2$
• Input sequence (sequence length = 2, each column vector in $\mathbb{R}^2$):

$X = \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}$

• Dimension of queries and keys: $d_k = 2$
• Projection matrices:

$W_Q = W_K = \begin{bmatrix} 2 & -1 \\ 1 & -1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 2 \end{bmatrix}$

Scaled Dot-Product Attention:

$\text{Scaled Dot-Product Attention}(Q, K, V) = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Scaled Dot-Product Attention

You are given a simple transformer attention setup with the following configuration:

• Embedding dimension: $d_{\text{model}} = 2$
• Input sequence (sequence length = 2, each column vector in $\mathbb{R}^2$):

$X = \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}$

• Dimension of queries and keys: $d_k = 2$
• Projection matrices:

$W_Q = W_K = \begin{bmatrix} 2 & -1 \\ 1 & -1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 2 \end{bmatrix}$

Scaled Dot-Product Attention:

$\text{Scaled Dot-Product Attention}(Q, K, V) = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Compute the final attention output:

$\text{Scaled Dot-Product Attention}(Q, K, V) = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

What is the sum of the values in the first row of the Scaled Dot-Product Attention output?

(Answer up to 2 decimal places.)

Consider an image $X$ of size $4 \times 4$, and a true label $y = 3$. The image is passed through a convolutional neural network with a single filter followed by a ReLU and a max-pooling operation.

$X = \begin{bmatrix} 1 & 0 & 3 & 2 \\ 0 & 1 & 0 & 2 \\ 2 & 1 & 1 & 0 \\ 1 & 0 & 2 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{bmatrix}$

Parameters:

• Convolution: stride $S = 2$, padding $P = 1$
• Max pooling: filter size $F = 2$, stride $S = 2$, padding $P = 0$

Based on the above data, answer the given subquestions.

What is the output of the convolution layer after applying ReLU?

Consider an image $X$ of size $4 \times 4$, and a true label $y = 3$. The image is passed through a convolutional neural network with a single filter followed by a ReLU and a max-pooling operation.

$X = \begin{bmatrix} 1 & 0 & 3 & 2 \\ 0 & 1 & 0 & 2 \\ 2 & 1 & 1 & 0 \\ 1 & 0 & 2 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{bmatrix}$

Parameters:

• Convolution: stride $S = 2$, padding $P = 1$
• Max pooling: filter size $F = 2$, stride $S = 2$, padding $P = 0$

Based on the above data, answer the given subquestions.

Consider an image $X$ of size $4 \times 4$, and a true label $y = 3$. The image is passed through a convolutional neural network with a single filter followed by a ReLU and a max-pooling operation.

$X = \begin{bmatrix} 1 & 0 & 3 & 2 \\ 0 & 1 & 0 & 2 \\ 2 & 1 & 1 & 0 \\ 1 & 0 & 2 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{bmatrix}$

Parameters:

• Convolution: stride $S = 2$, padding $P = 1$
• Max pooling: filter size $F = 2$, stride $S = 2$, padding $P = 0$

Based on the above data, answer the given subquestions.

Consider a CBOW model for learning word embeddings. The vocabulary is made up of three words, {good, bad, ugly}. $W$ and $C$ are the matrices that contain the word and context embeddings respectively. The columns in each matrix correspond to the embeddings. Both matrices are of shape $2 \times 3$:

$W = \begin{bmatrix} \text{good} & \text{bad} & \text{ugly} \\ 1 & 0 & -1 \\ -1 & 1 & 0 \end{bmatrix}, \quad C = \begin{bmatrix} \text{good} & \text{bad} & \text{ugly} \\ 1 & -1 & 1 \\ 1 & -1 & -1 \end{bmatrix}$

The context window is 1, meaning, the next word is predicted using just the current word as context. Recall that we use softmax to make predictions at the output.

Based on the above data, answer the given subquestions.

Consider a CBOW model for learning word embeddings. The vocabulary is made up of three words, {good, bad, ugly}. $W$ and $C$ are the matrices that contain the word and context embeddings respectively. The columns in each matrix correspond to the embeddings. Both matrices are of shape $2 \times 3$:

$W = \begin{bmatrix} \text{good} & \text{bad} & \text{ugly} \\ 1 & 0 & -1 \\ -1 & 1 & 0 \end{bmatrix}, \quad C = \begin{bmatrix} \text{good} & \text{bad} & \text{ugly} \\ 1 & -1 & 1 \\ 1 & -1 & -1 \end{bmatrix}$

The context window is 1, meaning, the next word is predicted using just the current word as context. Recall that we use softmax to make predictions at the output.

Based on the above data, answer the given subquestions.

The CBOW model is now used to generate a “sentence” or a string of words. First we pass the word “good” and retain the word with highest probability as the output, say word1, which is in turn passed as input to the model. If the model is run this way for exactly three time steps, what is the sentence that it outputs? Note that the sentence here is “word1 word2 word3”.

Consider a CBOW model for learning word embeddings. The vocabulary is made up of three words, {good, bad, ugly}. $W$ and $C$ are the matrices that contain the word and context embeddings respectively. The columns in each matrix correspond to the embeddings. Both matrices are of shape $2 \times 3$:

$W = \begin{bmatrix} \text{good} & \text{bad} & \text{ugly} \\ 1 & 0 & -1 \\ -1 & 1 & 0 \end{bmatrix}, \quad C = \begin{bmatrix} \text{good} & \text{bad} & \text{ugly} \\ 1 & -1 & 1 \\ 1 & -1 & -1 \end{bmatrix}$

The context window is 1, meaning, the next word is predicted using just the current word as context. Recall that we use softmax to make predictions at the output.

Based on the above data, answer the given subquestions.

Now consider updating the word embeddings using the sample “good good”. The first “good” in the string is used as context and the second “good” as the true label. Use cross entropy as the loss function and run one iteration of gradient descent with η = 1 starting with the existing values for the embeddings.
Find the updated word embedding for “good” and choose the most appropriate option from below. Note that you have to compute the updated word embedding for “good” and not its context embedding.

Based on the above data, answer the given subquestions.

Find s1.

Based on the above data, answer the given subquestions.

Find s2.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.