Large Language Models Quiz 1: 25 February 2024 (January 2024 term)

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme

$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

Assume the index for the position starts from zero. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme

$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

Assume the index for the position starts from zero. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme

$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

Assume the index for the position starts from zero. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$

Based on the above data, answer the given subquestions.

The bottommost encoder layer of the transformer contains the following parameter matrices

$W_Q = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0 \\ 1 & 0 \end{bmatrix} \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \quad W_O = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$

given that the embeddings are row vectors, compute the attention score to produce the new representation for the token ($T$) at the 3-rd position in the sequence $X$.