Quiz Space

Deep Learning · End Term · 1 Sept 2024 · May 2024 term

Question 6: Consider a Transformer model with the following specifica…

Question 6

+2 marksNumerical answer

Consider a Transformer model with the following specifications for the decoder part:

  • Input dimension (embedding size): 20
  • Number of heads in multi-head attention: 2
  • head output dimension: 10
  • Dimension of feed-forward network: 16
  • Number of layers in the decoder: 3

Assume that each decoder layer contains:

  • One multi-head attention mechanism for self-attention.
  • One multi-head attention mechanism for encoder-decoder attention.
  • One feed-forward network.
  • No bias terms are included.

Calculate the total number of parameters in the decoder part.

Show answer

Correct answer: 3680

See the whole paperTake it as a mock test

Question 6 of 25 in the IIT Madras BS Deep Learning (Deep Learning) End Term paper sat on 1 Sept 2024, in the May 2024 term (IIT M DEGREE AN EXAM QDB3 01 Sep 2024). It carries 2 marks.

More questions from this paper

  1. Q1Figure question
  2. Q2Consider a dataset with 150 samples and a batch size of 15. If each minibatch iteration contributes an average loss of …
  3. Q3If you use hierarchical softmax with a binary tree where each leaf node represents a word in the vocabulary, and the vo…
  4. Q4In a Skip-gram model with a vocabulary size V = 100, an embedding dimension D = 10, and a window size of 3 (on each sid…
  5. Q5Figure question
  6. Q7Consider a feedforward neural network with the following structure: One input layer with 2 nodes One hidden layer with …
  7. Q8A neural network has the following structure: Input Layer: \mathbf{h_0} = \mathbf{x}, where \mathbf{x} \in \mathbb{R}^{…
  8. Q9Given an input array X and a kernel/filter K as follows: X = \begin{bmatrix} -1 & -1 & 0 & 2 \ -2 & 1 & 0 & 0 \ 1 & -1 …
  9. Q10What is the derivative of the ReLU activation function at x = 10?
  10. Q11In terms of convergence speed, which gradient descent method can show the most rapid progress initially but may suffer …
  11. Q12How does the use of early stopping in training a neural network affect the model’s performance on unseen data?
  12. Q13Which of the following statements is/are not true with respect to a dropout rate of 0.2?
  13. Q14In the context of unsupervised pretraining of artificial neural networks, which of the following statements accurately …
  14. Q15What are the maximum values of the derivatives of sigmoid and tanh?
  15. Q16Figure question
  16. Q17Given a matrix A with dimensions p × q, which of the following statements is NOT true regarding the rank-k approximatio…
  17. Q18Consider an encoder-decoder model trained with a batch size of 64. Each input sequence has a length of 12 tokens, and e…
  18. Q19In an encoder-decoder model, what is the significance of the context vector?
  19. Q20Given the attention weights αt,1 = 0.3, αt,2 = 0.4, αt,3 = 0.3 and the corresponding encoder hidden states h1 = [2, 1, …
  20. Q21In the Transformer model, what is the purpose of the multi-head attention mechanism?
  21. Q22In the context of the Transformer model’s encoder-decoder architecture, which of the following statements are correct?
  22. Q23In a time series prediction task using a GRU (Gated Recurrent Unit) network, the GRU processes input sequences where ea…
  23. Q24In a time series prediction task using a GRU (Gated Recurrent Unit) network, the GRU processes input sequences where ea…
  24. Q25In a time series prediction task using a GRU (Gated Recurrent Unit) network, the GRU processes input sequences where ea…