Question 9: Suppose the input sentence is “tea you enjoy often”. Usin…

A vocabulary $\mathcal{V} = \{\text{tea}, \text{you}, \text{enjoy}, \text{often}\}$ is associated with the following embedding matrix:

$E = \begin{bmatrix} 0 & 1 \\ 1 & 1 \\ 1 & 0 \\ -1 & 1 \end{bmatrix}$

Words are indexed in the order shown above: “tea” (index 0), “you” (1), “enjoy” (2), and “often” (3). No positional encodings are used.

The parameters of the attention layer are:

$W_Q = \begin{bmatrix} 0.5 & 1 \\ 1 & 0.5 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 1 & 0.2 \\ 0.5 & 1 \end{bmatrix},$

$W_O = \begin{bmatrix} 0.5 & 0.5 \\ 1 & -1 \end{bmatrix}$

For the input sequence “you enjoy tea often”. The computed attention matrix $A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)$ is given as:

$A = \begin{bmatrix} 0.55 & 0.32 & 0.11 & 0.02 \\ 0.43 & 0.25 & 0.21 & 0.1 \\ 0.39 & 0.39 & 0.16 & 0.07 \\ 0.24 & 0.4 & 0.2 & 0.17 \end{bmatrix}$

Answer the given subquestions. If you believe the information is insufficient, enter -1 as your answer.

Suppose the input sentence is “tea you enjoy often”. Using the same matrices and processing method, what is the attention score (i.e., softmax entry from A) for the query word “tea” attending to key word “often”?