LLM End Term: 22 December 2024, Set QDB1 (September 2024 term)

The input embeddings for the words “learning”, “brings” and “joy” are $h_1 = [0.5, 0.25, 1]$, $h_2 = [0.1, 0.25, 0]$, and $h_3 = [0.1, 0.1, 0.9]$, respectively. Note that the embeddings are row vectors. The projection matrices are as follows

$W_Q = \begin{bmatrix} 1 & 1 \\ -1 & 1 \\ 0 & 1 \end{bmatrix} \quad W_K = \begin{bmatrix} 0 & 1 \\ 1 & 0 \\ 0 & 1 \end{bmatrix} \quad W_V = \begin{bmatrix} 0 & 0 \\ -1 & -1 \\ 1 & 1 \end{bmatrix}$

The following quantities are computed as

$Q = HW_Q \quad K = HW_K \quad V = HW_V$

Let $e_j$ denote the unnormalized attention score, $a_j$ denote the normalized attention score (ignore the scaling by $\sqrt{d_k}$) and $z_j$ denote the linear combination of the value vectors for the $j - th$ word.

Enter the value of the first element, i.e., with index (0,0) of the matrix containing the partial derivatives, $\frac{\partial a_3}{\partial e_3}$.

Choose the correct statements regarding an encoder-decoder transformer model:

Consider the following statements about increasing the size of the vocabulary