Question 15: Question Description Embedding dimension : d_{model} = 4…

Question Description

• Embedding dimension : $d_{model} = 4$
• Input sequence (sequence length $= 2$, $d_{model} = 4$) :

$X = \begin{bmatrix} 1 & 0 \\ 0 & 2 \\ 1 & 0 \\ 0 & 2 \end{bmatrix}$

• Number of heads: 2 (each head operates on 2 dimensions)
• For both the heads $d_K = d_Q = d_V = 2$
• Weight matrices for $W_Q, W_K, W_V$ first head :

$W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \end{bmatrix}$

• Weight matrices for $W_Q, W_K, W_V$ second head :

$W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 1 & 1 & 2 & 1 \end{bmatrix}$

• The output from both heads are concatenated, then projected by:

$W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \\ 0 & 0.5 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

$\text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Concatenate the outputs from both the attention heads, then apply the output projection matrix Wo to produce the final output of the multi-head attention mechanism. Select the correct result of this operation.