Deep Learning End Term: 1 September 2024 (May 2024 term)

Consider a dataset with 150 samples and a batch size of 15. If each minibatch iteration contributes an average loss of 0.4, what will be the total loss after 15 epochs?

If you use hierarchical softmax with a binary tree where each leaf node represents a word in the vocabulary, and the vocabulary size (V ) is 16000, how many binary classifiers are needed?

In a Skip-gram model with a vocabulary size V = 100, an embedding dimension D = 10, and a window size of 3 (on each side), using negative sampling with 5 negative samples per positive sample, what is the total number of parameters in the model?

Consider a Transformer model with the following specifications for the decoder part:

• Input dimension (embedding size): 20
• Number of heads in multi-head attention: 2
• head output dimension: 10
• Dimension of feed-forward network: 16
• Number of layers in the decoder: 3

Assume that each decoder layer contains:

• One multi-head attention mechanism for self-attention.
• One multi-head attention mechanism for encoder-decoder attention.
• One feed-forward network.
• No bias terms are included.

Calculate the total number of parameters in the decoder part.

Consider a feedforward neural network with the following structure:

One input layer with 2 nodes

One hidden layer with 2 nodes

One output layer with 1 node

All weights and biases are initialized to zero. The activation function used in the hidden layer is the Rectified Linear Unit (ReLU), and the output layer uses the Sigmoid activation for binary classification. The network is trained with a binary cross-entropy loss function.

Two training examples are given: 1. Input vector: [2, -3], true label: 1 2. Input vector: [-1, 1], true label: 0

What will be the value of the total binary cross-entropy loss given these two training examples?

A neural network has the following structure:

• Input Layer: $\mathbf{h_0} = \mathbf{x}$, where $\mathbf{x} \in \mathbb{R}^{100}$
• Hidden Layers: Two hidden layers ($\mathbf{h_1}$ and $\mathbf{h_2}$), each with 120 neurons, using the sigmoid activation function.
• Output Layer: $\mathbf{O}$ with 8 neurons, using the softmax activation function.

Assuming that all weights between layers $\mathbf{h_2}$ and $\mathbf{O}$ are initialized to 0.2, with no bias associated with any neuron, what would be the computed cross-entropy loss for a given single data point? If the provided information is insufficient, please enter $-1$.

Given an input array $X$ and a kernel/filter $K$ as follows:

$X = \begin{bmatrix} -1 & -1 & 0 & 2 \\ -2 & 1 & 0 & 0 \\ 1 & -1 & -1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

$K = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 1 & 0 \\ 0 & 1 & 1 \end{bmatrix}$

• Convolve the kernel $K$ over the input $X$ with a stride $s = 1$ and no padding to obtain matrix $A$.
• Apply average pooling on $A$ to produce matrix $B$.
• Pass $B$ through the sigmoid (logistic) function to get the final output $\hat{y}$.

Given that $\frac{\partial L}{\partial \hat{y}} = -1$, determine the value of $\frac{\partial L}{\partial K_{00}}$, where $K_{00}$ is the element of $K$ at index $(0, 0)$.

What is the derivative of the ReLU activation function at x = 10?

In terms of convergence speed, which gradient descent method can show the most rapid progress initially but may suffer from high variance in updates?

How does the use of early stopping in training a neural network affect the model’s performance on unseen data?

Which of the following statements is/are not true with respect to a dropout rate of 0.2?

In the context of unsupervised pretraining of artificial neural networks, which of the following statements accurately describes the role and benefits of using unsupervised pretraining techniques for initializing a neural network?

What are the maximum values of the derivatives of sigmoid and tanh?

Given a matrix A with dimensions p × q, which of the following statements is NOT true regarding the rank-k approximation of A obtained through Singular Value Decomposition (SVD)?

Consider an encoder-decoder model trained with a batch size of 64. Each input sequence has a length of 12 tokens, and each output sequence has a length of 18 tokens. How many computational steps do the encoder and decoder take per batch respectively during training?

In an encoder-decoder model, what is the significance of the context vector?

Given the attention weights αt,1 = 0.3, αt,2 = 0.4, αt,3 = 0.3 and the corresponding encoder hidden states h1 = [2, 1, 0], h2 = [1, 2, 1], h3 = [0, 1, 2], calculate the context vector ct.

In the Transformer model, what is the purpose of the multi-head attention mechanism?

In the context of the Transformer model’s encoder-decoder architecture, which of the following statements are correct?

In a time series prediction task using a GRU (Gated Recurrent Unit) network, the GRU processes input sequences where each input is represented by a 2-dimensional vector ($x_t \in \mathbb{R}^2$). The GRU uses the following formulas for the hidden state and output at time step $t$:

$\begin{aligned} i_t &= \sigma(W_i s_{t-1} + U_i x_t + b_i) \\ o_t &= \sigma(W_o s_{t-1} + U_o x_t + b_o) \\ \tilde{s}_t &= \tanh(Ux_t + W(o_t \odot s_{t-1}) + b) \\ s_t &= (1 - i_t) \odot s_{t-1} + i_t \odot \tilde{s}_t \\ \hat{y}_t &= Vh_t + c \end{aligned}$

where $\odot$ denotes element-wise multiplication. Assume that $h_t \in \mathbb{R}^3$ and $\hat{y}_t \in \mathbb{R}^2$.

Based on the above data, answer the given subquestions.

Given that the GRU processes sequences of length 6 (T = 6), what is the total number of parameters (including biases) in the network?

In a time series prediction task using a GRU (Gated Recurrent Unit) network, the GRU processes input sequences where each input is represented by a 2-dimensional vector ($x_t \in \mathbb{R}^2$). The GRU uses the following formulas for the hidden state and output at time step $t$:

$\begin{aligned} i_t &= \sigma(W_i s_{t-1} + U_i x_t + b_i) \\ o_t &= \sigma(W_o s_{t-1} + U_o x_t + b_o) \\ \tilde{s}_t &= \tanh(Ux_t + W(o_t \odot s_{t-1}) + b) \\ s_t &= (1 - i_t) \odot s_{t-1} + i_t \odot \tilde{s}_t \\ \hat{y}_t &= Vh_t + c \end{aligned}$

where $\odot$ denotes element-wise multiplication. Assume that $h_t \in \mathbb{R}^3$ and $\hat{y}_t \in \mathbb{R}^2$.

Based on the above data, answer the given subquestions.

In a time series prediction task using a GRU (Gated Recurrent Unit) network, the GRU processes input sequences where each input is represented by a 2-dimensional vector ($x_t \in \mathbb{R}^2$). The GRU uses the following formulas for the hidden state and output at time step $t$:

$\begin{aligned} i_t &= \sigma(W_i s_{t-1} + U_i x_t + b_i) \\ o_t &= \sigma(W_o s_{t-1} + U_o x_t + b_o) \\ \tilde{s}_t &= \tanh(Ux_t + W(o_t \odot s_{t-1}) + b) \\ s_t &= (1 - i_t) \odot s_{t-1} + i_t \odot \tilde{s}_t \\ \hat{y}_t &= Vh_t + c \end{aligned}$

where $\odot$ denotes element-wise multiplication. Assume that $h_t \in \mathbb{R}^3$ and $\hat{y}_t \in \mathbb{R}^2$.

Based on the above data, answer the given subquestions.