Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer. - Source vocabulary size = 100 - Target vocabulary size = 200 - maximum sequence length = 32 - length of context window (T) for both encoder and decoder = 32 - number of heads n_h = 4 - dff = 4 dmodel - dq = dk = dv = \frac{dmodel}{n_h} Based on the above data, answer the given subquestions. How many parameters are there in the multi-head attention layer of the encoder (exclude the parameters in the WO matrix used for linear transformation and FFN layer)?

Large Language Models · Quiz 1 · 23 Feb 2025 · January 2025 term

Question 9: Consider the following configuration for the Vannila tran…

Question 9

+3 marksNumerical answer

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

Source vocabulary size $= 100$
Target vocabulary size $= 200$
maximum sequence length $= 32$
length of context window $(T)$ for both encoder and decoder $= 32$
number of heads $n_h = 4$
$dff = 4 * dmodel$
$dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

How many parameters are there in the multi-head attention layer of the encoder (exclude the parameters in the WO matrix used for linear transformation and FFN layer)?

Show answer

Correct answer: 3072

See the whole paper Take it as a mock test

Question 9 of 18 in the IIT Madras BS Large Language Models (LLM) Quiz 1 paper sat on 23 Feb 2025, in the January 2025 term (IIT M DEGREE AN EXAM QDB2 23 Feb 2025). It carries 3 marks.

Question 9: Consider the following configuration for the Vannila tran…

Question 9

More questions from this paper