Large Language Models Quiz 1: 27 October 2024 (September 2024 term)

Teacher forcing for training an transformer model is:

Choose the correct statements regarding transformer architecture:

Based on the above data, answer the given subquestions.

Assume the model has two layers (N = 2). Calculate the total number of parameters in the model (excluding the embedding layer and output layer). Moreover, no bias was added to the neuron in the FFNN layers.

Based on the above data, answer the given subquestions.

How many parameters does the output layer have?

Based on the above data, answer the given subquestions.

How many parameters does the embedding layer have?

Based on the above data, answer the given subquestions.

At time step t = 1, the prediction probability for the words “Naan”, “transformer” and “padaththai” are 0.55,0.15 and 0.2, respectively. What is the probability for the word “rasithen”? If you think, the given information is insufficient enter -1

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 7 & 11 \\ -3 & 5 & -2 \end{bmatrix}$

It has 3 samples with 3 features each i.e. batch size is 3. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1$, $\beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Say matrix F is computed by applying batch normalization on X, what will be sum of every element in first/topmost row of F?

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 7 & 11 \\ -3 & 5 & -2 \end{bmatrix}$

It has 3 samples with 3 features each i.e. batch size is 3. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1$, $\beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Say matrix F is computed by applying layer normalization on X, what will be sum of every element in first/leftmost column of F?

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 7 & 11 \\ -3 & 5 & -2 \end{bmatrix}$

It has 3 samples with 3 features each i.e. batch size is 3. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1$, $\beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 7 & 11 \\ -3 & 5 & -2 \end{bmatrix}$

It has 3 samples with 3 features each i.e. batch size is 3. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1$, $\beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V}$,

$\mathcal{V} =$([start], breaking, ground, a, is, seismology, study, experience, earth quake,[end]).

Assume that we have a pre-trained GPT model for text generation and the first input token to the model is always the special token [start]. The prediction probabilities outputted by the model are given below.

$\hat{Y} = \begin{bmatrix} 0.04 & 0.01 & 0.14 & 0.07 & 0.09 & 0.41 & 0.08 & 0.05 & 0.08 & 0.01 \\ 0.10 & 0.03 & 0.06 & 0.16 & 0.43 & 0.01 & 0.06 & 0.09 & 0.01 & 0.04 \\ 0.06 & 0.08 & 0.28 & 0.29 & 0.03 & 0.08 & 0.04 & 0.08 & 0.04 & 0.04 \\ 0.11 & 0.03 & 0.48 & 0.02 & 0.01 & 0.07 & 0.06 & 0.14 & 0.07 & 0.01 \\ 0.09 & 0.29 & 0.04 & 0.01 & 0.04 & 0.03 & 0.25 & 0.05 & 0.15 & 0.06 \\ 0.14 & 0.22 & 0.13 & 0.08 & 0.01 & 0.03 & 0.06 & 0.23 & 0.08 & 0.03 \end{bmatrix}$

The columns of the matrix are in the same order as the words listed in the vocabulary set above

The zeroth row of the matrix is the output probability distribution by the model given the special [Start] token as input. Following the Greedy Search decoding strategy, the subsequent rows give the conditional probability distribution conditioned over the previous tokens. If you think the given information is insufficient for any sub-questions, then enter -1 as your answer

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V}$,

$\mathcal{V} =$([start], breaking, ground, a, is, seismology, study, experience, earth quake,[end]).

Assume that we have a pre-trained GPT model for text generation and the first input token to the model is always the special token [start]. The prediction probabilities outputted by the model are given below.

$\hat{Y} = \begin{bmatrix} 0.04 & 0.01 & 0.14 & 0.07 & 0.09 & 0.41 & 0.08 & 0.05 & 0.08 & 0.01 \\ 0.10 & 0.03 & 0.06 & 0.16 & 0.43 & 0.01 & 0.06 & 0.09 & 0.01 & 0.04 \\ 0.06 & 0.08 & 0.28 & 0.29 & 0.03 & 0.08 & 0.04 & 0.08 & 0.04 & 0.04 \\ 0.11 & 0.03 & 0.48 & 0.02 & 0.01 & 0.07 & 0.06 & 0.14 & 0.07 & 0.01 \\ 0.09 & 0.29 & 0.04 & 0.01 & 0.04 & 0.03 & 0.25 & 0.05 & 0.15 & 0.06 \\ 0.14 & 0.22 & 0.13 & 0.08 & 0.01 & 0.03 & 0.06 & 0.23 & 0.08 & 0.03 \end{bmatrix}$

The columns of the matrix are in the same order as the words listed in the vocabulary set above

The zeroth row of the matrix is the output probability distribution by the model given the special [Start] token as input. Following the Greedy Search decoding strategy, the subsequent rows give the conditional probability distribution conditioned over the previous tokens. If you think the given information is insufficient for any sub-questions, then enter -1 as your answer

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V}$,

$\mathcal{V} =$([start], breaking, ground, a, is, seismology, study, experience, earth quake,[end]).

Assume that we have a pre-trained GPT model for text generation and the first input token to the model is always the special token [start]. The prediction probabilities outputted by the model are given below.

$\hat{Y} = \begin{bmatrix} 0.04 & 0.01 & 0.14 & 0.07 & 0.09 & 0.41 & 0.08 & 0.05 & 0.08 & 0.01 \\ 0.10 & 0.03 & 0.06 & 0.16 & 0.43 & 0.01 & 0.06 & 0.09 & 0.01 & 0.04 \\ 0.06 & 0.08 & 0.28 & 0.29 & 0.03 & 0.08 & 0.04 & 0.08 & 0.04 & 0.04 \\ 0.11 & 0.03 & 0.48 & 0.02 & 0.01 & 0.07 & 0.06 & 0.14 & 0.07 & 0.01 \\ 0.09 & 0.29 & 0.04 & 0.01 & 0.04 & 0.03 & 0.25 & 0.05 & 0.15 & 0.06 \\ 0.14 & 0.22 & 0.13 & 0.08 & 0.01 & 0.03 & 0.06 & 0.23 & 0.08 & 0.03 \end{bmatrix}$

The columns of the matrix are in the same order as the words listed in the vocabulary set above

The zeroth row of the matrix is the output probability distribution by the model given the special [Start] token as input. Following the Greedy Search decoding strategy, the subsequent rows give the conditional probability distribution conditioned over the previous tokens. If you think the given information is insufficient for any sub-questions, then enter -1 as your answer

Based on the above data, answer the given subquestions.

Consider a vocabulary $\mathcal{V}$,

$\mathcal{V} =$([start], breaking, ground, a, is, seismology, study, experience, earth quake,[end]).

Assume that we have a pre-trained GPT model for text generation and the first input token to the model is always the special token [start]. The prediction probabilities outputted by the model are given below.

$\hat{Y} = \begin{bmatrix} 0.04 & 0.01 & 0.14 & 0.07 & 0.09 & 0.41 & 0.08 & 0.05 & 0.08 & 0.01 \\ 0.10 & 0.03 & 0.06 & 0.16 & 0.43 & 0.01 & 0.06 & 0.09 & 0.01 & 0.04 \\ 0.06 & 0.08 & 0.28 & 0.29 & 0.03 & 0.08 & 0.04 & 0.08 & 0.04 & 0.04 \\ 0.11 & 0.03 & 0.48 & 0.02 & 0.01 & 0.07 & 0.06 & 0.14 & 0.07 & 0.01 \\ 0.09 & 0.29 & 0.04 & 0.01 & 0.04 & 0.03 & 0.25 & 0.05 & 0.15 & 0.06 \\ 0.14 & 0.22 & 0.13 & 0.08 & 0.01 & 0.03 & 0.06 & 0.23 & 0.08 & 0.03 \end{bmatrix}$

The columns of the matrix are in the same order as the words listed in the vocabulary set above

The zeroth row of the matrix is the output probability distribution by the model given the special [Start] token as input. Following the Greedy Search decoding strategy, the subsequent rows give the conditional probability distribution conditioned over the previous tokens. If you think the given information is insufficient for any sub-questions, then enter -1 as your answer

Based on the above data, answer the given subquestions.

Suppose we use Top$-k$ sampling, with $k = 3$, what is the probability of the predicted word equal to “Seismology” given the “[Start]“ token as the initial input $P(y_1 = Seismology|y_0 = [Start])$?

Rajesh has created a GPT-like transformer model. However he doesn't have access to large compute infrastructure, so he has configured the original architecture in the following manner:

1. He has taken a vocabulary of size 1000 words/tokens.
2. Sequence length is 64.
3. Embedding dimension and $d_{model}$ is 128.
4. There are only 4 transformer blocks (layers).
5. Each transformer block (layer) has only 4 attention heads.
6. FFN hidden layer size is 256.

Based on the above data, answer the given subquestions.

What will be the shape of positional embedding?

Rajesh has created a GPT-like transformer model. However he doesn't have access to large compute infrastructure, so he has configured the original architecture in the following manner:

1. He has taken a vocabulary of size 1000 words/tokens.
2. Sequence length is 64.
3. Embedding dimension and $d_{model}$ is 128.
4. There are only 4 transformer blocks (layers).
5. Each transformer block (layer) has only 4 attention heads.
6. FFN hidden layer size is 256.

Based on the above data, answer the given subquestions.

Rajesh has created a GPT-like transformer model. However he doesn't have access to large compute infrastructure, so he has configured the original architecture in the following manner:

1. He has taken a vocabulary of size 1000 words/tokens.
2. Sequence length is 64.
3. Embedding dimension and $d_{model}$ is 128.
4. There are only 4 transformer blocks (layers).
5. Each transformer block (layer) has only 4 attention heads.
6. FFN hidden layer size is 256.

Based on the above data, answer the given subquestions.

The input embeddings for the words “learning”, “brings” and “joy” are $h_1 = [0.5, 0.25, 1]$, $h_2 = [0.1, 0.25, 0]$, and $h_3 = [0.1, 0.1, 0.9]$, respectively. Note that the embeddings are row vectors. The projection matrices are as follows

$W_Q = \begin{bmatrix} 1 & 1 \\ -1 & 1 \\ 0 & 1 \end{bmatrix} \quad W_K = \begin{bmatrix} 0 & 1 \\ 1 & 0 \\ 0 & 1 \end{bmatrix} \quad W_V = \begin{bmatrix} 0 & 0 \\ -1 & -1 \\ 1 & 1 \end{bmatrix}$

The following quantities are computed as

$Q = HW_Q \quad K = HW_K \quad V = HW_V$

Let $e_j$ denote the unnormalized attention score, $a_j$ denote the normalized attention score (ignore the scaling by $\sqrt{d_k}$) and $z_j$ denote the linear combination of the value vectors for the $j - th$ word.

Suppose the gradient vector $\frac{\partial L}{\partial z_3} = [1, 2]$, then what is the gradient vector $\frac{\partial L}{\partial e_3}$? Enter the sum of gradients.

Consider the embedding vector for a word, $x = [0.1, 0.2, -0.3, 0.4]$. Suppose the word is at position 2 in the given sentence. Add the corresponding position embedding $p$ to the word embedding to get $h$, i.e. the sum of the elements in $h = x + p$. Use the fixed-sinusoidal position embedding vector calculated using the formula given below

$PE(pos, 2i) = sin\left(\frac{pos}{10000^{2i/dmodel}}\right)$

$PE(pos, 2i+1) = cos\left(\frac{pos}{10000^{2i/dmodel}}\right)$

What is $h[0] + h[1]$ i.e. sum of first two elements of $h$?