Question 8: For the input “you enjoy tea often”, compute the final re…

A vocabulary $\mathcal{V} = \{\text{tea}, \text{you}, \text{enjoy}, \text{often}\}$ is associated with the following embedding matrix:

$E = \begin{bmatrix} 0 & 1 \\ 1 & 1 \\ 1 & 0 \\ -1 & 1 \end{bmatrix}$

Words are indexed in the order shown above: “tea” (index 0), “you” (1), “enjoy” (2), and “often” (3). No positional encodings are used.

The parameters of the attention layer are:

$W_Q = \begin{bmatrix} 0.5 & 1 \\ 1 & 0.5 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 1 & 0.2 \\ 0.5 & 1 \end{bmatrix},$

$W_O = \begin{bmatrix} 0.5 & 0.5 \\ 1 & -1 \end{bmatrix}$

For the input sequence “you enjoy tea often”. The computed attention matrix $A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)$ is given as:

$A = \begin{bmatrix} 0.55 & 0.32 & 0.11 & 0.02 \\ 0.43 & 0.25 & 0.21 & 0.1 \\ 0.39 & 0.39 & 0.16 & 0.07 \\ 0.24 & 0.4 & 0.2 & 0.17 \end{bmatrix}$

Answer the given subquestions. If you believe the information is insufficient, enter -1 as your answer.

For the input “you enjoy tea often”, compute the final representation of the word “enjoy” after the attention layer (i.e., after applying WQ, WK, WV , attention weights, and WO). Enter the sum of the elements in the resulting vector.