Question 13: Question Description Embedding dimension : d_{model} = 4…

Question Description

• Embedding dimension : $d_{model} = 4$
• Input sequence (sequence length $= 2$, $d_{model} = 4$) :

$X = \begin{bmatrix} 1 & 0 \\ 0 & 2 \\ 1 & 0 \\ 0 & 2 \end{bmatrix}$

• Number of heads: 2 (each head operates on 2 dimensions)
• For both the heads $d_K = d_Q = d_V = 2$
• Weight matrices for $W_Q, W_K, W_V$ first head :

$W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \end{bmatrix}$

• Weight matrices for $W_Q, W_K, W_V$ second head :

$W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 1 & 1 & 2 & 1 \end{bmatrix}$

• The output from both heads are concatenated, then projected by:

$W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \\ 0 & 0.5 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

$\text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Select the scaled dot-product attention for the first head: