Scaled Dot-Product Attention
You are given a simple transformer attention setup with the following configuration:
Embedding dimension: d model = 2 d_{\text{model}} = 2 d model = 2
Input sequence (sequence length = 2, each column vector in R 2 \mathbb{R}^2 R 2
X = [ 1 1 1 0 ] X = \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix} X = [ 1 1 1 0 ]
Dimension of queries and keys: d k = 2 d_k = 2 d k = 2
Projection matrices:
W Q = W K = [ 2 − 1 1 − 1 ] , W V = [ 0 1 1 2 ] W_Q = W_K = \begin{bmatrix} 2 & -1 \\ 1 & -1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 2 \end{bmatrix} W Q = W K = [ 2 1 − 1 − 1 ] , W V = [ 0 1 1 2 ]
Scaled Dot-Product Attention:
Scaled Dot-Product Attention ( Q , K , V ) = softmax ( Q T K d k ) V T \text{Scaled Dot-Product Attention}(Q, K, V) = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T Scaled Dot-Product Attention ( Q , K , V ) = softmax ( d k Q T K ) V T
Based on the above data, answer the given subquestions.
