Large Language Models Quiz 1: 25 February 2024 (January 2024 term)

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme

$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

Assume the index for the position starts from zero. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme

$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

Assume the index for the position starts from zero. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme

$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

Assume the index for the position starts from zero. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$

Based on the above data, answer the given subquestions.

The bottommost encoder layer of the transformer contains the following parameter matrices

$W_Q = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0 \\ 1 & 0 \end{bmatrix} \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \quad W_O = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$

given that the embeddings are row vectors, compute the attention score to produce the new representation for the token ($T$) at the 3-rd position in the sequence $X$.

Consider a vocabulary $\mathcal{V} = (A, T, C, G)$ and the corresponding embedding matrix $E = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}$. For example, the $0 - th$ row of $E$ is the embedding for $0 - th$ token of $\mathcal{V}$ and so on. The following sequence $X = (G, C, C, T, A, G)$ is to be processed by the encoder layer of the transformer architecture by appropriately adding the positional information using the following encoding scheme

$P(pos, 2i) = sin\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

$P(pos, 2i+1) = cos\left(\frac{pos}{10^{(2i/dmodel)}}\right)$

Assume the index for the position starts from zero. Add positional information to all the tokens in the sequence. Let the resultant sequence be denoted by $H = E(X) + P$, ($H$ will be used in the subsequent questions,please write it down).Here, $E(X)$ denotes the embedding for each token in $X$

Based on the above data, answer the given subquestions.

The bottommost encoder layer of the transformer contains the following parameter matrices

$W_Q = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0 \\ 1 & 0 \end{bmatrix} \quad W_V = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \quad W_O = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$

given that the embeddings are row vectors, compute the attention score to produce the new representation for the token ($T$) at the 3-rd position in the sequence $X$.

Which token in the sequence was given the highest score?

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

• Source and target vocabulary size $= 100$
• maximum sequence length $= 32$
• length of context window ($T$) for both encoder and decoder $= 32$
• number of heads $n_h = 2$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

Suppose the number of learnable parameters in the source input embedding layer is 1600, how many parameters are there in the positional embedding layer of the source language? Assume the positional embeddings are learnable.

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

• Source and target vocabulary size $= 100$
• maximum sequence length $= 32$
• length of context window ($T$) for both encoder and decoder $= 32$
• number of heads $n_h = 2$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

How many parameters are there in the multi-head attention layer of the encoder (exclude the parameters in the WO matrix used for linear transformation)?

Suppose we run the pre-trained GPT model in an autoregressive fashion for generating a text sequence. Then, the statement that “ We do not need to add the positional embedding to the predicted tokens at every time step” is

Suppose we have a dataset for machine translation tasks with thousands of samples. Suppose a team considers training the transformer model. The model could be trained in two approaches A  Autoregressive training
B  Teacher forcing
Choose the correct statements

A team trained a Language model using the GPT architecture using a large corpus of text. The model configurations are given below

• Vocabulary size $= 30000$
• $dmodel = 512$
• length of context window $T = 1024$
• $dff = 4 * dmodel$
• $dq = dk = dv = 64$

They trained the model for millions of steps and stopped the training after the loss had decreased significantly. Then, they used the model to generate the text by giving the special starting token [BOS] as an input to the model. Select the length of the text sequence that the model could generate.

Choose the correct statements

Consider a GPT model used for Causal language modelling. We feed the input sentence “This is a cool idea” to the model by appending special staring [BOS] and ending [EOS] tokens ( that is, “[BOS] This is a cool idea [EOS]”). Assume the context length of the model is 7. The attention matrix computed in one of the attention layers is given below

Table 1: Attention score

Based on the above data, answer the given subquestions.

The attention score matrix given in Table 1 is appropriate for the causal language modelling task

Consider a GPT model used for Causal language modelling. We feed the input sentence “This is a cool idea” to the model by appending special staring [BOS] and ending [EOS] tokens ( that is, “[BOS] This is a cool idea [EOS]”). Assume the context length of the model is 7. The attention matrix computed in one of the attention layers is given below

Table 1: Attention score

Based on the above data, answer the given subquestions.

Assume the time step starts from t = 0 and ends at t = 6. Suppose the model is at time step t = 4, what is the attention value assigned for the word “is”?

The statement that “the Next Sentence Prediction (NSP) task requires the BERT model to run autoregressively given the first sentence( or segment)” is