Deep Learning End Term: 24 December 2023, Set ADB3 (September 2023 term)

Suppose that a neural network has millions of parameters (weights and biases). A team decides to use an optimization algorithm with a learning rate scheme that is local to each parameter in the network. Moreover, the learning rate changes in each iteration such that it should decrease on the steep surface and increase on the gentle surface. Which of the following optimization algorithms satisfy the team’s requirements?

Suppose we divide the available training samples into mini batches of size 32 to train a model with mini-batch gradient descent. Assume that we have 33 different machines to train the model. One out of 33 machines acts as a master machine. The actual weight update happens in the master machine. The master machine can send one sample for the rest of the machines along with a copy of the model in its current state to compute the gradients. We call this entire set-up parallelization. Which of the following deep learning architectures can be trained in parallel then?

Select all correct statements about the bias and variance of deep learning models.

Based on the above data, answer the given subquestions.

How many neurons are there in the network? Every neuron in the network has bias associated with it?
Note: A neuron is a computation unit that takes in some inputs and produces an output.

Based on the above data, answer the given subquestions.

How many parameters (including bias) does the network have to learn? Assume dropout regularization is applied.

Based on the above data, answer the given subquestions.

What is the size of the vocabulary |V |?

Based on the above data, answer the given subquestions.

Suppose we consider the three words (wood,woodchuck, much). Assume we use one-hot encoded vector representation for all these words. The statement that,“The cosine similarity between the pair (wood,woodchuck) is greater than the pair (wood,much)” is

Suppose that we need to develop an RNN model for sentiment classification tasks. The input to the model is a sentence composed of 10 words and the output is the sentiment (positive or negative). Assume that each word is represented as a vector of length $100 \times 1$ and the output labels are one-hot encoded. Further, the state vector $s_t$ and the prediction $\hat{y}_t$ are computed as follows

$s_t = \sigma(Ux_t + Ws_{t-1} + b)$

$\hat{y}_t = \mathcal{O}(Vs_t + c)$

The state vector $s_t$ is initialized with all zeros of size $50 \times 1$.

Based on the above data, answer the given subquestions.

How many parameters (including bias) are there in the network?

Suppose that we need to develop an RNN model for sentiment classification tasks. The input to the model is a sentence composed of 10 words and the output is the sentiment (positive or negative). Assume that each word is represented as a vector of length $100 \times 1$ and the output labels are one-hot encoded. Further, the state vector $s_t$ and the prediction $\hat{y}_t$ are computed as follows

$s_t = \sigma(Ux_t + Ws_{t-1} + b)$

$\hat{y}_t = \mathcal{O}(Vs_t + c)$

The state vector $s_t$ is initialized with all zeros of size $50 \times 1$.

Based on the above data, answer the given subquestions.

For the given input sentence containing 10 words, how many sequential time steps does RNN take to make a final prediction?

Suppose that we need to develop an RNN model for sentiment classification tasks. The input to the model is a sentence composed of 10 words and the output is the sentiment (positive or negative). Assume that each word is represented as a vector of length $100 \times 1$ and the output labels are one-hot encoded. Further, the state vector $s_t$ and the prediction $\hat{y}_t$ are computed as follows

$s_t = \sigma(Ux_t + Ws_{t-1} + b)$

$\hat{y}_t = \mathcal{O}(Vs_t + c)$

The state vector $s_t$ is initialized with all zeros of size $50 \times 1$.

Based on the above data, answer the given subquestions.

Suppose we train the model using the BPTT algorithm for 200 iterations. In each iteration, we feed the input sentence and make a prediction, compute the loss, back-propagate through time and update the parameter. How many times does the parameter matrix W get updated over 200 iterations?

Consider an intermediate feature map $H$ obtained after applying convolution operation on the input $X$ using kernel $F$.

$H = \begin{bmatrix} 1 & 2 & 0 & 0 & -1 & 1 \\ 3 & 0 & 2 & 1 & 3 & 0 \\ 1 & 1 & 0 & 0 & 1 & 0 \\ 1 & -1 & 1 & 2 & 0 & 0 \\ 4 & 0 & 0 & 3 & -2 & 1 \end{bmatrix}$

Apply the max-pooling operation with stride $s = 2$ and no padding ($p = 0$) and store the resultant output in matrix $H_m$. The prediction $\hat{y}$ is simply the sum of elements in $H_m$. Suppose $\frac{\partial L}{\partial \hat{y}} = 1.5$ (that is, the gradient of loss with respect to the prediction). What is the gradient $\frac{\partial L}{\partial H_{10}}$ where $H_{10}$ is the element at the 1-st row and 0-th column? If you think the given info is insufficient, enter -1 as the answer.