Deep Learning End Term: 22 December 2024 (September 2024 term)

You are given a dataset of $10 \times 10$ grayscale images. Your goal is to build a 5-class classifier. You have to adopt one of the following two options:

• Model A: the input is flattened into a 100-dimensional vector, followed by a fully-connected layer with 5 neurons without bias
• Model B: the input is directly given to a convolutional layer with five $10 \times 10$ filters

Suppose you make your choice on the basis of the number of parameters in the models, $p_A$ = number of parameters in ModelA, and similarly let $p_B$ = number of parameters in ModelB.

In the context of RNNs, what structural feature of LSTMs helps reduce the impact of vanishing gradients?

As per gradient descent, we should move towards 180 degrees with respect to gradient direction. What will happen if we move between 90 and 180 degrees? Consider the loss function to be convex.

Which of the following statements about the parameter β in RMSProp is correct?

Consider a feed-forward neural network containing three inputs and one output. It consists of 50 hidden layers, each with two neurons. The activation function used in all the hidden layers is ReLU, the output layer uses the Sigmoid activation function for binary classification. The network is trained with a binary cross-entropy loss function.
The network architecture is as follows:
- Input layer: 3 nodes
- Hidden layer: 2 nodes each
- Output layer: 1 node
Assume all weights are initialized to 1 and all biases to 0. For an input of [-10, 5, -20], what will be the output of the neural network?

In the transformer model, how are self-attention weights calculated for a given token?

Consider a sigmoid neuron that takes in an input vector $\mathbf{x} = \begin{bmatrix} 1 \\ 0.5 \\ 1 \end{bmatrix}$. The weight vector $\mathbf{w}$ is initialized to $\begin{bmatrix} 0.5 \\ 1 \\ 0.5 \end{bmatrix}$ and $\mathbf{b} = 1$. The output from the sigmoid neuron is

$z = \mathbf{w}^\mathbf{T}\mathbf{x} + \mathbf{b}$

$\hat{\text{y}} = \frac{1}{1 + exp^{-(z)}}$

Suppose we use the following loss function

$\mathcal{L} = \frac{1}{2}(y - \hat{y})^2$

Based on the above data, answer the given subquestions.

Consider a sigmoid neuron that takes in an input vector $\mathbf{x} = \begin{bmatrix} 1 \\ 0.5 \\ 1 \end{bmatrix}$. The weight vector $\mathbf{w}$ is initialized to $\begin{bmatrix} 0.5 \\ 1 \\ 0.5 \end{bmatrix}$ and $\mathbf{b} = 1$. The output from the sigmoid neuron is

$z = \mathbf{w}^\mathbf{T}\mathbf{x} + \mathbf{b}$

$\hat{\text{y}} = \frac{1}{1 + exp^{-(z)}}$

Suppose we use the following loss function

$\mathcal{L} = \frac{1}{2}(y - \hat{y})^2$

Based on the above data, answer the given subquestions.

Consider a sigmoid neuron that takes in an input vector $\mathbf{x} = \begin{bmatrix} 1 \\ 0.5 \\ 1 \end{bmatrix}$. The weight vector $\mathbf{w}$ is initialized to $\begin{bmatrix} 0.5 \\ 1 \\ 0.5 \end{bmatrix}$ and $\mathbf{b} = 1$. The output from the sigmoid neuron is

$z = \mathbf{w}^\mathbf{T}\mathbf{x} + \mathbf{b}$

$\hat{\text{y}} = \frac{1}{1 + exp^{-(z)}}$

Suppose we use the following loss function

$\mathcal{L} = \frac{1}{2}(y - \hat{y})^2$

Based on the above data, answer the given subquestions.

Which of the following techniques does NOT help prevent a model from overfitting?

Suppose you have a vocabulary of 5,000 unique words and you want to train a CBOW model with a window size of 3 (on each side) and with an embedding dimension of 100. How many parameters (weights) will the embedding layer have?

Assume that your CBOW model outputs a probability distribution over a vocabulary of 20,000 words for a given context. If the correct target word is word number 150, and the model’s predicted probability for this word is 0.2, calculate the cross-entropy loss for this prediction. (use natural log)

In a Skip-gram model with a window size of 2 (on each side), how many unique pairs of target and context words will be generated for the following sentence:
‘Predicting the future is not magic, it is artificial intelligence’
Note: Ignore the punctuations in your calculation

Based on the above data, answer the given subquestions.

With a total of 7 time steps (T = 7), implying prediction for a word of length 6), what is the total count of parameters (including bias) within the network?

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

If all the parameters (including bias) in the network are initialized to zero, what will be the total loss after 10 time steps (assume that indices start with 1) for the input $[0,0,1,0,0,0]^T$? The ground truth ($y \in \mathbb{R}^5$) for each time step is given by the following sequence $[y_0, y_4, y_1, y_2, y_3, y_4, y_4, y_4, y_2, y_3]$. Assume the loss to be cross-entropy at each time step. (Use natural log and write the answer correct up to two decimal places.) each $y$ is one-hot encoder (i.e $y_0$means $= [1,0,0,0,0]^T$ )

In the context of the Transformer model’s encoder-decoder architecture, which of the following statements are accurate?

Consider a Transformer model with the following specifications for the decoder part: • Input dimension (embedding size): 16
• Number of heads in multi-head attention: 2
• Head output dimension: 8
• Dimension of the feed-forward network: 32
• Number of layers in the decoder: 2
Note: The final output from the attention heads is obtained by concatenating the outputs from the individual heads and hence does not cost any additional parameters.
Assume that each decoder layer contains:
• One multi-head attention mechanism for self-attention.
• One multi-head attention mechanism for encoder-decoder attention.
• One feed-forward network.
• No bias terms are included.
Calculate the total number of parameters in the decoder part.