Question 5: Consider the following configuration for the Vannila tran…

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

• Source and target vocabulary size $= 100$
• maximum sequence length $= 32$
• length of context window ($T$) for both encoder and decoder $= 32$
• number of heads $n_h = 2$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

Suppose the number of learnable parameters in the source input embedding layer is 1600, how many parameters are there in the positional embedding layer of the source language? Assume the positional embeddings are learnable.