Deep Learning End Term: 28 April 2024, Set QDB3 (January 2024 term)

Consider the MP neuron model and its applicability to representing boolean functions. Select the correct statements:

How many sigmoid neurons do we require to construct a tower function using single hidden layer to approximate a 2 dimensional continuous function ?

Consider a feedforward neural network with one hidden layer trained using backpropagation for a binary classification task. The network has the following architecture:

• Input layer with 15 neurons
• Hidden layer with 25 neurons
• Output layer with 1 neuron

During the backpropagation process, the derivative of the sigmoid activation function $\sigma(z)$ with respect to its argument $z$ is given by:

$\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))$

If the loss function used for binary classification is the binary cross-entropy loss, and the activation fuction at hidden layer and output layer is sigmoid. The output of the neural network is denoted as $\hat{y}$, and the true label is denoted as $y$, what is the expression for $\frac{\partial L}{\partial w_j}$, where $w_j$ represents the weights connecting the $j$th neuron of hidden layer to the output layer? Assume that the output of $j$th neuron of hidden layer is $h_j$ and no biases in the network.

In the context of mini-batch gradient descent, if reducing the size of the minibatch to half makes your model take thrice as many epochs to reach convergence, how does this affect the total number of parameter updates compared to using the original mini-batch size? Assume everything else remains constant.

Which of the following statements about AdaMax optimisation algorithm are True?

Choose all the correct options from below.

Consider an ensemble consisting of 5 models, where each model has an individual error rate of 50% on the test set. Assuming a simple majority voting scheme, what is the probability that a given instance is misclassified by the ensemble?
Assume that the models are independent and their errors are uncorrelated.

Given the true outputs for a dataset, y = [1,2,3,4], and the predictions from three different models (A, B, and C) trained on different subsets of the training data, as follows:
Model A predictions: A = [1.1,1.9,3.1,3.9]
Model B predictions: B = [0.9,2.1,2.9,4.1]
Model C predictions: C = [2,2,3,4]
Calculate the average variance for these models evaluated in a regression task. Assume that each model uses the same parameters but is trained on different subsets of the training data. (Enter your answer up to 3 decimal places.)

Consider a Convolutional Neural Network (CNN) architecture for image classification with the following layers:
1. Convolutional layer with 24 filters of size 3 × 3, with a stride of 1 and no padding.
2. Max pooling layer with a pool size of 2 × 2 and a stride of 2.
3. Convolutional layer with 56 filters of size 4 × 4, with a stride of 1 and no padding.
4. Max pooling layer with a pool size of 2 × 2 and a stride of 2.
5. Fully connected layer with 108 neurons.
6. Output layer with 10 neurons (for 10 classes) using softmax activation.
If the input image size is 84 × 84 × 3, and the network has no bias term, how many parameters are there in the CNN?

Suppose you have a vocabulary of 5,000 unique words and you want to train a CBOW model with a window size of 3 (on each side) and with an embedding dimension of 500. How many parameters (weights) will the embedding layer have?

Assume that your CBOW model outputs a probability distribution over a vocabulary of 10,000 words for a given context. If the correct target word is word number 200, and the model’s predicted probability for this word is 0.3, calculate the cross-entropy loss for this prediction.

In a Skip-gram model with a window size of 2 (on each side), how many unique pairs of target and context words will be generated for the following sentence:
‘Finish easy questions as early as possible’

In a Skip-gram model with a vocabulary of 20000 words, if you choose a negative sampling rate of 5 negative samples for each positive sample, how many total samples will be generated for a sentence with 10 words and a window size of 2 (on each side)?

Given a scenario where there are 8 possible letters represented as one-hot encoded vectors of length 8, the RNN employs the following formulas for the state vector and output at time step $t$:

$\begin{aligned} s_t &= \sigma(Ux_i + Ws_{t-1} + b) \\ \hat{y}_t &= O(Vs_t + c) \end{aligned}$

Here, $\sigma$ and $O$ denote the sigmoid and softmax functions, respectively. Assume that $s_t \in \mathbb{R}^2$ and $y_t \in \mathbb{R}^8$

Based on the above data, answer the given subquestions.

With a total of 20 time steps (T = 20, implying prediction for a word of length 19), what is the total count of parameters (including bias) within the network?

Given a scenario where there are 8 possible letters represented as one-hot encoded vectors of length 8, the RNN employs the following formulas for the state vector and output at time step $t$:

$\begin{aligned} s_t &= \sigma(Ux_i + Ws_{t-1} + b) \\ \hat{y}_t &= O(Vs_t + c) \end{aligned}$

Here, $\sigma$ and $O$ denote the sigmoid and softmax functions, respectively. Assume that $s_t \in \mathbb{R}^2$ and $y_t \in \mathbb{R}^8$

Based on the above data, answer the given subquestions.

Given a scenario where there are 8 possible letters represented as one-hot encoded vectors of length 8, the RNN employs the following formulas for the state vector and output at time step $t$:

$\begin{aligned} s_t &= \sigma(Ux_i + Ws_{t-1} + b) \\ \hat{y}_t &= O(Vs_t + c) \end{aligned}$

Here, $\sigma$ and $O$ denote the sigmoid and softmax functions, respectively. Assume that $s_t \in \mathbb{R}^2$ and $y_t \in \mathbb{R}^8$

Based on the above data, answer the given subquestions.

If all the parameters (including bias) in the network are initialized to zero, what will be the total loss after 20 time steps (assume that indices start with 1) for the input [0, 0, 0, 1, 0, 0, 0, 0]^(T) ? assume the loss to be cross-entropy at each time step. Write your answer correct to two decimal places.

Consider training a Recurrent Neural Network (RNN) for a sequence prediction task. Which of the following statements accurately describe challenges in training RNNs?

State True or False. GRUs are popular variant of LSTMs in which output and forget gates are combined.