Deep Learning End Term: 28 April 2024, Set QDB1 (January 2024 term)

Consider a scenario where you have a dataset with overlapping classes (that is instances from different classes share similar or identical feature values), and you decide to train a perceptron model for classification.
Assertion (A): The perceptron model may struggle to classify instances accurately when classes overlap in the feature space.
Reason (R): The perceptron learning algorithm aims to find a linear decision boundary that separates the classes, and in the presence of overlapping classes, it may not be able to capture the underlying patterns effectively.
Select the correct option:

Consider a feedforward neural network with one hidden layer trained using backpropagation for a binary classification task. The network has the following architecture:

• Input layer with 15 neurons
• Hidden layer with 25 neurons
• Output layer with 1 neuron

During the backpropagation process, the derivative of the sigmoid activation function $\sigma(z)$ with respect to its argument $z$ is given by:

$\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))$

If the loss function used for binary classification is the binary cross-entropy loss, and the activation fuction at hidden layer and output layer is sigmoid. The output of the neural network is denoted as $\hat{y}$, and the true label is denoted as $y$, what is the expression for $\frac{\partial L}{\partial w_j}$, where $w_j$ represents the weights connecting the $j$th neuron of hidden layer to the output layer? Assume that the output of $j$th neuron of hidden layer is $h_j$ and no biases in the network.

In the context of mini-batch gradient descent, if doubling the size of the mini-batch makes your model take twice as many epochs to reach convergence, how does this affect the total number of parameter updates compared to using the original mini-batch size? Assume everything else remains constant.

Suppose you are working with a sparse dataset and using Momentum-based Gradient Descent (GD) and AdaGrad algorithms for optimization. Which of the following statements accurately describes the behavior of weight and bias term updates?

State True or False. Sigmoid activation function helps mitigating vanishing gradient problem better than ReLU activation function.

How many weight matrices does a LSTM unit and GRU unit learns respectively during backpropagation?

Which statements about LSTM and GRU units in handling the problem of exploding/vanishing gradients are correct?

Consider an ensemble consisting of 3 models, where each model has an individual error rate of 50% on the test set. Assuming a simple majority voting scheme, what is the probability that a given instance is misclassified by the ensemble?
Assume that the models are independent and their errors are uncorrelated.

Given the true outputs for a dataset, y = [1,2,3,4], and the predictions from three different models (A, B, and C) trained on different subsets of the training data, as follows:
Model A predictions: A = [1.1,1.9,3.1,3.9]
Model B predictions: B = [0.9,2.1,2.9,4.1]
Model C predictions: C = [2,2,3,4]
Calculate the squared bias (bias²) for these models evaluated in a regression task. Assume that each model uses the same parameters but is trained on different subsets of the training data. (Enter your answer up to 3 decimal places.)

Consider a Convolutional Neural Network (CNN) architecture for image classification with the following layers:
1. Convolutional layer with 32 filters of size 3 × 3, with a stride of 1 and no padding.
2. Max pooling layer with a pool size of 2 × 2 and a stride of 2.
3. Convolutional layer with 64 filters of size 4 × 4, with a stride of 1 and no padding.
4. Max pooling layer with a pool size of 2 × 2 and a stride of 2.
5. Fully connected layer with 128 neurons.
6. Output layer with 10 neurons (for 10 classes) using softmax activation.
If the input image size is 64 × 64 × 3, and the network has no bias term, how many parameters are there in the CNN?

Suppose you have a vocabulary of 10,000 unique words and you want to train a CBOW model with a window size of 2 (on each side) and with an embedding dimension of 300. How many parameters (weights) will the embedding layer have?

Assume that your CBOW model outputs a probability distribution over a vocabulary of 20,000 words for a given context. If the correct target word is word number 150, and the model’s predicted probability for this word is 0.02, calculate the cross-entropy loss for this prediction.

In a Skip-gram model with a window size of 2 (on each side), how many unique pairs of target and context words will be generated for the following sentence:
‘The idea is to die young as late as possible’

In a Skip-gram model with a vocabulary of 15000 words, if you choose a negative sampling rate of 10 negative samples for each positive sample, how many total samples will be generated for a sentence with 8 unique words and a window size of 3 (on each side)?

Given a scenario where there are 5 possible letters represented as one-hot encoded vectors of length 5, the RNN employs the following formulas for the state vector and output at time step $t$:

$\begin{aligned} s_t &= \sigma(Ux_i + Ws_{t-1} + b) \\ \hat{y}_t &= O(Vs_t + c) \end{aligned}$

Here, $\sigma$ and $O$ denote the sigmoid and softmax functions, respectively. Assume that $s_t \in \mathbb{R}^2$ and $y_t \in \mathbb{R}^5$

Based on the above data, answer the given subquestions.

With a total of 10 time steps (T = 10, implying prediction for a word of length 9), what is the total count of parameters (including bias) within the network?

Given a scenario where there are 5 possible letters represented as one-hot encoded vectors of length 5, the RNN employs the following formulas for the state vector and output at time step $t$:

$\begin{aligned} s_t &= \sigma(Ux_i + Ws_{t-1} + b) \\ \hat{y}_t &= O(Vs_t + c) \end{aligned}$

Here, $\sigma$ and $O$ denote the sigmoid and softmax functions, respectively. Assume that $s_t \in \mathbb{R}^2$ and $y_t \in \mathbb{R}^5$

Based on the above data, answer the given subquestions.

Given a scenario where there are 5 possible letters represented as one-hot encoded vectors of length 5, the RNN employs the following formulas for the state vector and output at time step $t$:

$\begin{aligned} s_t &= \sigma(Ux_i + Ws_{t-1} + b) \\ \hat{y}_t &= O(Vs_t + c) \end{aligned}$

Here, $\sigma$ and $O$ denote the sigmoid and softmax functions, respectively. Assume that $s_t \in \mathbb{R}^2$ and $y_t \in \mathbb{R}^5$

Based on the above data, answer the given subquestions.

If all the parameters (including bias) in the network are initialized to zero, what will be the total loss after 10 time steps (assume that indices start with 1) for the input [0, 0, 1, 0, 0]^(T) ? assume the loss to be cross-entropy at each time step. Write your answer correct to two decimal places.

Select the correct statement regarding the vanishing and exploding gradient problem in RNN.