Deep Learning End Term: 3 September 2023, Set QPE1-S2 (May 2023 term)

Consider the statement “the attention mechanism in RNN based Encoder- Decoder architecture helps the decoder to understand the context of words in a given sentence”. The statement is

Suppose that we have a deep Feed Forward Fully Connected Neural Network. The network is observed to have a high variance. Then, which of the following techniques regularize the parameter of the network to reduce the high variance?

The logistic sigmoid function is defined as follows,

$f(x) = \frac{1}{1 + exp(-(wx + b))}$

The parameters are initialized to $w = 0.5$ $b = 0.5$. Suppose the loss in defined as

$L = \frac{1}{2}(f(x) - y)^2$

where $y$ is the true value. Compute the gradient of $w$ for the following sample $x = 1, y = 1$.

Consider a simple RNN for a binary sequence classification problem.

Based on the above data, answer the given subquestions.

How many neurons are there in the network. Every neuron in the network has bias associated with it?
Note: A neuron is a computation unit that takes in some inputs and produce an output.

Based on the above data, answer the given subquestions.

How many learnable parameters (including bias) does the network have? Assume dropout regularization is applied.

Based on the above data, answer the given subquestions.

The statement that, in general, the activation function of neurons in the network are not necessarily be the same across the network is

Consider a sentence inside the quote “I may be wrong, and you may be right, and by an effort, we may get nearer to the truth”
Based on the above data, answer the given subquestions.

Consider a sentence inside the quote “I may be wrong, and you may be right, and by an effort, we may get nearer to the truth”
Based on the above data, answer the given subquestions.

Suppose all words in the vocabulary are represented using one-hot-encoded vector of size $|V|$. Then compute the ordered pair-wise (that is, Cartesian product of $V \times V$) cosine similarity between two word representations and enter their sum.

The update rule for the ADAM (Adaptive Moments) optimization algorithm is given below,

$\begin{aligned} m_t &= \beta_1 m_{t-1} + (1 - \beta_1)\nabla w_t \\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2)(\nabla w_t)^2 \\ w_{t+1} &= w_t - \frac{\eta}{\sqrt{\hat{v}_t + \epsilon}}\hat{m}_t \end{aligned}$

Here, $0 \le \beta_1 < 1$ and $0 \le \beta_2 < 1$ and $t$ starts from zero (that is, $t = 0, 1, 2, \cdots$). Both $m_t$ and $v_t$ are initialized to zero. However, the update rule uses the bias corrected version of $m_t$ and $v_t$. Which of the following is the bias corrected version of $m_t$?

Helper:

$m_t = (1 - \beta_1)\sum_{\tau=0}^{t} \beta_1^{t-\tau} \nabla w_\tau$

and assume that $E[\nabla w_\tau] = E[\nabla w] \quad \forall \tau$, if required.

Based on the above data, answer the given subquestions.

Suppose the input is $x$ (one hot representation of context word) and the corresponding label is $y$ (one hot representation of target word). The quantities $h, u_c, \hat{y}$ are computed as follows,

$h = u_c = W_{context}\, x, \quad z = W_{word}\, u_c$

$\hat{y} = softmax(z)$

Choose the expression that the model has to minimize using cross entropy loss (Assume natural logarithm where required).

Based on the above data, answer the given subquestions.