Question 1

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme $$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$$ $$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$$ Assume the index for the **position starts from zero**. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$ Based on the above data, answer the given subquestions. Figure from the original question paper Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme $$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$$ $$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$$ Assume the index for the **position starts from zero**. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$ Based on the above data, answer the given subquestions. Figure from the original question paper Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme $$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$$ $$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$$ Assume the index for the **position starts from zero**. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$ Based on the above data, answer the given subquestions. The bottommost encoder layer of the transformer contains the following parameter matrices $$W_Q = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0 \\ 1 & 0 \end{bmatrix} \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \quad W_O = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$$ given that the embeddings are row vectors, compute the attention score to produce the new representation for the token ($T$) at the 3-rd position in the sequence $X$. Figure from the original question paper