Question Description - Embedding dimension : d_{model} = 4 - Input sequence (sequence length = 2, d_{model} = 4) : X = \begin{bmatrix} 1 & 0 \ 0 & 2 \ 1 & 0 \ 0 & 2 \end{bmatrix} - Number of heads: 2 (each head operates on 2 dimensions) - For both the heads d_K = d_Q = d_V = 2 - Weight matrices for W_Q, W_K, W_V first head : W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \ 2 & 2 & 2 & 2 \end{bmatrix} - Weight matrices for W_Q, W_K, W_V second head : W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \ 1 & 1 & 2 & 1 \end{bmatrix} - The output from both heads are concatenated, then projected by: W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \ 0 & 0.5 & 0 & 0 \ 0 & 0 & 1 & 0 \ 0 & 0 & 0 & 1 \end{bmatrix} \text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T Based on the above data, answer the given subquestions. Select the scaled dot-product attention for the first head:

(B) Option figure from the original question paper

(A) Option figure from the original question paper

(C) Option figure from the original question paper

(D) Option figure from the original question paper

Question Description - Embedding dimension : d_{model} = 4 - Input sequence (sequence length = 2, d_{model} = 4) : X = \begin{bmatrix} 1 & 0 \ 0 & 2 \ 1 & 0 \ 0 & 2 \end{bmatrix} - Number of heads: 2 (each head operates on 2 dimensions) - For both the heads d_K = d_Q = d_V = 2 - Weight matrices for W_Q, W_K, W_V first head : W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \ 2 & 2 & 2 & 2 \end{bmatrix} - Weight matrices for W_Q, W_K, W_V second head : W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \ 1 & 1 & 2 & 1 \end{bmatrix} - The output from both heads are concatenated, then projected by: W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \ 0 & 0.5 & 0 & 0 \ 0 & 0 & 1 & 0 \ 0 & 0 & 0 & 1 \end{bmatrix} \text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T Based on the above data, answer the given subquestions. Select the scaled dot-product attention for the second head:

(A) Option figure from the original question paper

(B) Option figure from the original question paper

(C) Option figure from the original question paper

(D) Option figure from the original question paper

Question Description - Embedding dimension : d_{model} = 4 - Input sequence (sequence length = 2, d_{model} = 4) : X = \begin{bmatrix} 1 & 0 \ 0 & 2 \ 1 & 0 \ 0 & 2 \end{bmatrix} - Number of heads: 2 (each head operates on 2 dimensions) - For both the heads d_K = d_Q = d_V = 2 - Weight matrices for W_Q, W_K, W_V first head : W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \ 2 & 2 & 2 & 2 \end{bmatrix} - Weight matrices for W_Q, W_K, W_V second head : W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \ 1 & 1 & 2 & 1 \end{bmatrix} - The output from both heads are concatenated, then projected by: W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \ 0 & 0.5 & 0 & 0 \ 0 & 0 & 1 & 0 \ 0 & 0 & 0 & 1 \end{bmatrix} \text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T Based on the above data, answer the given subquestions. Concatenate the outputs from both the attention heads, then apply the output projection matrix Wo to produce the final output of the multi-head attention mechanism. Select the correct result of this operation.

(D) Option figure from the original question paper

(A) Option figure from the original question paper

(B) Option figure from the original question paper

(C) Option figure from the original question paper

LLM Quiz 1 PYQ May 2025: 13 Jul 2025 · IITM BS

Question 1

+2 marksOne correct option

Transformers process input tokens:

A
One at a time (sequentially)
B
In reverse order
C
All at once (in parallel)
D
Only after seeing the full input

Show answer

Correct answer

C
All at once (in parallel)

Question 1 on its own page

Question 2

+2 marksOne correct option

What is the purpose of the softmax function in the attention mechanism?

A
Normalize attention scores to a probability distribution
B
Add non-linearity to the model
C
To predict the correct class for the loss function
D
Remove redundant features from the input

Show answer

Correct answer

A
Normalize attention scores to a probability distribution

Question 2 on its own page

Question 3

+2 marksOne correct option

What is the main difference between GPT and BERT pre-training objectives?

A
GPT uses Masked Language Modeling, BERT uses Causal Language Modeling
B
GPT uses Causal Language Modeling, BERT uses Masked Language Modeling
C
Both use Masked Language Modeling
D
Both use Causal Language Modeling

Show answer

Correct answer

B
GPT uses Causal Language Modeling, BERT uses Masked Language Modeling

Question 3 on its own page

Question 4

+2 marksOne correct option

In Top-K sampling for language generation, increasing the value of K typically has which of the following effects?

A
It makes the output more deterministic and repetitive.
B
It reduces the probability of selecting high-frequency words.
C
It increases the diversity of the generated text but may reduce coherence if K is too large.
D
It guarantees grammatical correctness by focusing on top-ranked tokens only.

Show answer

Correct answer

C
It increases the diversity of the generated text but may reduce coherence if K is too large.

Question 4 on its own page

Question 5

+2 marksOne correct option

Why are residual connections important in transformer architectures?

A
They reduce memory consumption.
B
They add extra cost of compute by adding batch normalization.
C
They help in training deep networks by enabling gradient flow.
D
They remove the need for layer normalization.

Show answer

Correct answer

C
They help in training deep networks by enabling gradient flow.

Question 5 on its own page

Question 6

+3 marksOne or more correct options

Which of the following statements are true regarding causal language modeling (CLM)?

Select all that apply.

A
The model only attends to past and current tokens during training.
B
The model is trained by predicting the next token in a sequence.
C
The model uses bidirectional context.
D
The CLM objective is commonly used for encoder-only models.

Show answer

Correct answers

A
The model only attends to past and current tokens during training.
B
The model is trained by predicting the next token in a sequence.

Question 6 on its own page

Question 7

+3 marksOne or more correct options

Which of the following are valid reasons why transformer-based large language models are widely used in natural language processing?

Select all that apply.

A
Transformers process input sequences in parallel, enabling faster training.
B
They use recurrence to remember long-term dependencies more effectively than LSTMs.
C
Transformers are limited to short text inputs due to their architecture.
D
Large transformer models can be fine-tuned for various NLP tasks using a single pre-trained model.

Show answer

Correct answers

A
Transformers process input sequences in parallel, enabling faster training.
D
Large transformer models can be fine-tuned for various NLP tasks using a single pre-trained model.

Question 7 on its own page

Question 8

+3 marksOne or more correct options

Which elements are included in BERT’s input representation for Next Sentence Prediction?

Select all that apply.

A
B
C
D

Show answer

Correct answers

A
B
C

Question 8 on its own page

Question 9

+3 marksOne or more correct options

How does Top-p (nucleus) sampling differ from Top-K sampling in language generation?

Select all that apply.

A
It samples only from a fixed number of tokens at each step.
B
It samples from the smallest set of tokens whose cumulative probability exceeds p.
C
It always selects the top-p tokens with equal probability.
D
It guarantees diversity by selecting all low-probability tokens.

Show answer

Correct answer

B
It samples from the smallest set of tokens whose cumulative probability exceeds p.

Question 9 on its own page

Question 10

+3 marksNumerical answer

Consider the embedding vector for a word, $x = [0.5, -0.4, 0.3, -0.2]^T$ . Suppose the word is at position 3 in the given sentence. Add the corresponding position embedding $p$ to the word embedding to get $h$ , i.e. the sum of the elements in $h = x + p$ . Use the fixed sinusoidal position embedding vector calculated using the formula given below

$PE(pos, 2i) = sin\left(\frac{pos}{10000^{2i/dmodel}}\right)$

$PE(pos, 2i+1) = cos\left(\frac{pos}{10000^{2i/dmodel}}\right)$

What is the value of $h[0] - h[1]$ i.e. substract second element from the first element of $h$ vector ? (Answer correct up to 2 values after the decimal)

Show answer

Correct answer: 2.03 (accepted within ±0.05)

Question 10 on its own page

Question 11

+3 marksNumerical answer

Suppose a language model outputs the following logits for a vocabulary of size 4:

Token	Logit
the	2.0
sky	1.0
is	0.5
blue	$-1.0$

You apply temperature scaling with temperature $T = 0.5$ . After applying the softmax on the temperature-scaled logits, what is the probability assigned to the token “the”? (Give your answer up to 2 decimal places)

Show answer

Correct answer: 0.84 (accepted within ±0.03)

Question 11 on its own page

Question 12

+3 marksNumerical answer

Show answer

Correct answer: 1 (accepted within ±0.1)

Question 12 on its own page

Question 13

+3 marksOne correct option

Question Description

Embedding dimension : $d_{model} = 4$
Input sequence (sequence length $= 2$ , $d_{model} = 4$ ) :

$X = \begin{bmatrix} 1 & 0 \\ 0 & 2 \\ 1 & 0 \\ 0 & 2 \end{bmatrix}$

Number of heads: 2 (each head operates on 2 dimensions)
For both the heads $d_K = d_Q = d_V = 2$
Weight matrices for $W_Q, W_K, W_V$ first head :

$W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \end{bmatrix}$

Weight matrices for $W_Q, W_K, W_V$ second head :

$W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 1 & 1 & 2 & 1 \end{bmatrix}$

The output from both heads are concatenated, then projected by:

$W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \\ 0 & 0.5 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

$\text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Select the scaled dot-product attention for the first head:

A
B
C
D

Show answer

Correct answer

B

Question 13 on its own page

Question 14

+3 marksOne correct option

Question Description

Embedding dimension : $d_{model} = 4$
Input sequence (sequence length $= 2$ , $d_{model} = 4$ ) :

$X = \begin{bmatrix} 1 & 0 \\ 0 & 2 \\ 1 & 0 \\ 0 & 2 \end{bmatrix}$

Number of heads: 2 (each head operates on 2 dimensions)
For both the heads $d_K = d_Q = d_V = 2$
Weight matrices for $W_Q, W_K, W_V$ first head :

$W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \end{bmatrix}$

Weight matrices for $W_Q, W_K, W_V$ second head :

$W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 1 & 1 & 2 & 1 \end{bmatrix}$

The output from both heads are concatenated, then projected by:

$W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \\ 0 & 0.5 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

$\text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Select the scaled dot-product attention for the second head:

A
B
C
D

Show answer

Correct answer

A

Question 14 on its own page

Question 15

+2 marksOne correct option

Question Description

Embedding dimension : $d_{model} = 4$
Input sequence (sequence length $= 2$ , $d_{model} = 4$ ) :

$X = \begin{bmatrix} 1 & 0 \\ 0 & 2 \\ 1 & 0 \\ 0 & 2 \end{bmatrix}$

Number of heads: 2 (each head operates on 2 dimensions)
For both the heads $d_K = d_Q = d_V = 2$
Weight matrices for $W_Q, W_K, W_V$ first head :

$W_Q^1, W_K^1, W_V^1 = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \end{bmatrix}$

Weight matrices for $W_Q, W_K, W_V$ second head :

$W_Q^2, W_K^2, W_V^2 = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 1 & 1 & 2 & 1 \end{bmatrix}$

The output from both heads are concatenated, then projected by:

$W_O = \begin{bmatrix} 0.5 & 0 & 0 & 0 \\ 0 & 0.5 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}$

$\text{Scaled Dot-Product Attention (Q, K, V )} = \text{softmax}\left(\frac{Q^T K}{\sqrt{d_k}}\right) V^T$

Based on the above data, answer the given subquestions.

Concatenate the outputs from both the attention heads, then apply the output projection matrix Wo to produce the final output of the multi-head attention mechanism. Select the correct result of this operation.

A
B
C
D

Show answer

Correct answer

D

Question 15 on its own page

Question 16

+1 markNumerical answer

Based on the above data, answer the given subquestions.

For the given input matrix X and multihead attention output MHA(X), apply a residual connection and store the result in matrix R, and finally compute the sum of all elements in matrix R.

Show answer

Correct answer: 36

Question 16 on its own page

Question 17

+2 marksNumerical answer

Based on the above data, answer the given subquestions.

Show answer

Correct answer: 8

Question 17 on its own page

Question 18

+3 marksNumerical answer

Based on the above data, answer the given subquestions.

Show answer

Correct answer: 8

Question 18 on its own page

Question 19

+2 marksNumerical answer

The table presents the conditional probability distribution over vocabulary tokens at each timestep during sequence generation. Each column corresponds to a timestep (from 1 to 5), and the values represent the probability of selecting each token given the tokens chosen in all previous timesteps. For timestep t, the values in the column represent:

$P(\text{token}_t \mid \text{token}_1, \text{token}_2, \ldots, \text{token}_{t-1})$

timesteps

	1	2	3	4	5
the	0.07	0.19	0.20	0.42	0.08
is	0.04	0.13	0.12	0.26	0.08
sky	0.23	0.12	0.35	0.05	0.15
today	0.03	0.18	0.16	0.16	0.46
very	0.22	0.01	0.07	0.11	0.15
blue	0.33	0.25	0.08	0.05	0.08

Table 1: Probability distribution over vocabulary tokens at each timestep

Based on the above data, answer the given subquestions.

In exhaustive search, at timestep t=1, we run the decoder once to obtain probability distributions over all tokens in the vocabulary. Given the 6 tokens available in the table in the main question, how many times must we run the decoder at timestep t=4?

Show answer

Correct answer: 216

Question 19 on its own page

Question 20

+1 markNumerical answer

The table presents the conditional probability distribution over vocabulary tokens at each timestep during sequence generation. Each column corresponds to a timestep (from 1 to 5), and the values represent the probability of selecting each token given the tokens chosen in all previous timesteps. For timestep t, the values in the column represent:

$P(\text{token}_t \mid \text{token}_1, \text{token}_2, \ldots, \text{token}_{t-1})$

timesteps

	1	2	3	4	5
the	0.07	0.19	0.20	0.42	0.08
is	0.04	0.13	0.12	0.26	0.08
sky	0.23	0.12	0.35	0.05	0.15
today	0.03	0.18	0.16	0.16	0.46
very	0.22	0.01	0.07	0.11	0.15
blue	0.33	0.25	0.08	0.05	0.08

Table 1: Probability distribution over vocabulary tokens at each timestep

Based on the above data, answer the given subquestions.

How many total sequences of exactly length 5 are possible according to exhaustive search?

Show answer

Correct answer: 7776

Question 20 on its own page

Question 21

+2 marksNumerical answer

The table presents the conditional probability distribution over vocabulary tokens at each timestep during sequence generation. Each column corresponds to a timestep (from 1 to 5), and the values represent the probability of selecting each token given the tokens chosen in all previous timesteps. For timestep t, the values in the column represent:

$P(\text{token}_t \mid \text{token}_1, \text{token}_2, \ldots, \text{token}_{t-1})$

timesteps

	1	2	3	4	5
the	0.07	0.19	0.20	0.42	0.08
is	0.04	0.13	0.12	0.26	0.08
sky	0.23	0.12	0.35	0.05	0.15
today	0.03	0.18	0.16	0.16	0.46
very	0.22	0.01	0.07	0.11	0.15
blue	0.33	0.25	0.08	0.05	0.08

Table 1: Probability distribution over vocabulary tokens at each timestep

Based on the above data, answer the given subquestions.

If we use top-k sampling with k=2 at timestep 1, what is the normalized probability of selecting token “sky” at the timestep=1?

Show answer

Correct answer: 0.41

Question 21 on its own page