Question 6: Consider the following configuration for the Vannila tran…

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

• Source and target vocabulary size $= 100$
• maximum sequence length $= 32$
• length of context window ($T$) for both encoder and decoder $= 32$
• number of heads $n_h = 2$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

How many parameters are there in the multi-head attention layer of the encoder (exclude the parameters in the WO matrix used for linear transformation)?