Figure from the original question paper

(D) To avoid numerical issues and loss of gradients during training.

(A) To increase the numerical precision of the attention scores.

(B) To reduce the overall computational cost of the attention mechanism.

(C) To ensure that the sum of the attention scores for each query equals 1.

Figure from the original question paper

(C) Option figure from the original question paper

(A) Option figure from the original question paper

(B) Option figure from the original question paper

(D) Option figure from the original question paper

(E) Option figure from the original question paper

Figure from the original question paper

(C) Option figure from the original question paper

(A) Option figure from the original question paper

(B) Option figure from the original question paper

(D) Option figure from the original question paper

You have a mini-batch of 2 samples, each with 2 features: X = \begin{bmatrix} 1 & 2 \ 3 & 4 \end{bmatrix} - Rows represent samples - Columns represent features If Batch Normalization is applied across the batch for each feature (assume \gamma = 1, \beta = 0 and \epsilon = 0.), what is the resulting normalized mini-batch?

(A) Option figure from the original question paper

(B) Option figure from the original question paper

(C) Option figure from the original question paper

(D) Option figure from the original question paper

Figure from the original question paper

(C) Option figure from the original question paper

(A) Option figure from the original question paper

(B) Option figure from the original question paper

(D) Option figure from the original question paper

(E) Option figure from the original question paper

A transformer decoder uses a vocabulary of size 5. At step 1, the predicted token probabilities are: | Token | Probability | |---|---| | A | 0.40 | | B | 0.30 | | C | 0.20 | | D | 0.05 | | E | 0.05 | Which statement(s) is/are correct?

(B) Beam search with beam size 2 will keep both A and B whereas Top-2 sampling will keep either A or B.

(D) Beam search with beam size 2 will always keep the top-2 most probable tokens, whereas Top-2 sampling will restrict choices to top-2 tokens and then sample one probabilistically.

(A) Beam search with beam size 2 will keep either A or B as per probability whereas Top-2 sampling will keep both A and B.

(C) Beam search with beam size 2 will randomly choose any two tokens, whereas Top-2 sampling will always pick the two least probable tokens.

You are analyzing a GPT-style decoder-only Transformer trained as a causal language model with the following setup: - Sequence length: T = 1024 tokens per input sequence. - Batch size: B = 32. - Hidden dimension: d_{\text{model}} = 768. - Number of attention heads: h = 8. - d_v = \frac{d_{\text{model}}}{h} - Training uses next-token prediction with teacher forcing and a causal mask. - The input sequence does not include a beginning-of-sequence (BOS) token. Answer the given sub-questions: During autoregressive inference with GPT:

(B) Each new token is generated conditioned on previously generated tokens.

(D) Beam search or other decoding strategies are commonly used to decide the next token.

(A) The model can generate all tokens in parallel.

(C) Teacher forcing is used to stabilize outputs.

September 2025 term · Large Language Models · BSDA5004

Large Language Models Quiz 1: 26 October 2025 (September 2025 term)

The IIT Madras BS Large Language Models (LLM) Quiz 1 paper sat on 26 Oct 2025, in the September 2025 term: 20 questions for 50 marks in 120 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.

Questions: 20
Marks: 50
Duration: 120 min
MCQ: 9
MSQ: 5
Numerical: 6

Take as mock test Practise with answers

Updated 25 Sept 2026

Official paper: IIT M DEGREE AN EXAM QDB2 26 Oct 2025 · No negative marking.

Question 1

+2 marksOne correct option

A
To increase the numerical precision of the attention scores.
B
To reduce the overall computational cost of the attention mechanism.
C
To ensure that the sum of the attention scores for each query equals 1.
D
To avoid numerical issues and loss of gradients during training.

Show answer

Correct answer

D
To avoid numerical issues and loss of gradients during training.

Question 1 on its own page

Question 2

+2 marksOne correct option

Which of the following is true regarding sinusoidal encoding?

A
The encoding vector of words present at even positions in the given text sequence uses sine, and the encoding vector of words present at odd positions uses cosine.
B
The encoding vector of words present at odd positions in the given text sequence uses sine, and the encoding vector of words present at even positions uses cosine.
C
The encoding vector for each word position contains values computed using both sine (for even dimensions) and cosine (for odd dimensions).

Show answer

Correct answer

C
The encoding vector for each word position contains values computed using both sine (for even dimensions) and cosine (for odd dimensions).

Question 2 on its own page

Question 3

+2 marksOne correct option

We have a transformer model translating English to French. Suppose the target sequence is:

Target (French): [‘je’, ‘suis’, ‘heureux’]

Suppose the model predicted at time step t:

t = 1: ‘je’

t = 2: ‘content’

t = 3: ‘heureux’

If teacher forcing is used, what will be the decoder input at time step 3?

A
‘content’
B
‘je’
C
‘suis’
D
‘heureux’

Show answer

Correct answer

C
‘suis’

Question 3 on its own page

Question 4

+2 marksOne correct option

Show answer

Correct answer

Question 4 on its own page

Question 5

+2 marksOne correct option

In the context of language model training, teacher forcing is a technique where:

A
The correct target tokens from the training data are used as input for the next time step, regardless of the model’s prediction.
B
The model’s predictions are used as the input for the next time step, even if they are incorrect.
C
The model is forced to learn without any pre-trained weights.
D
The model is forced to correct the input data.

Show answer

Correct answer

A
The correct target tokens from the training data are used as input for the next time step, regardless of the model’s prediction.

Question 5 on its own page

Question 6

+2 marksOne correct option

Show answer

Correct answer

Question 6 on its own page

Question 7

+3 marksOne correct option

You have a mini-batch of 2 samples, each with 2 features:

$X = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$

Rows represent samples
Columns represent features

If Batch Normalization is applied across the batch for each feature (assume $\gamma = 1, \beta = 0$ and $\epsilon = 0.$ ), what is the resulting normalized mini-batch?

Show answer

Correct answer

Question 7 on its own page

Question 8

+3 marksOne correct option

Show answer

Correct answer

Question 8 on its own page

Question 9

+3 marksOne correct option

For a vocabulary size of 10, how many beams will there be in greedy search, beam search with beam size 3, and exhaustive search at step 1 and step 3, respectively?

A
Step 1:
Greedy: 1, Beam: 1, Exhaustive: 1
Step 3:
Greedy: 1, Beam: 1, Exhaustive: 1
B
Step 1:
Greedy: 1, Beam: 3, Exhaustive: 10
Step 3:
Greedy: 1, Beam: 3, Exhaustive: 10
C
Step 1:
Greedy: 1, Beam: 3, Exhaustive: 1
Step 3:
Greedy: 1, Beam: 3, Exhaustive: 100
D
Step 1:
Greedy: 1, Beam: 3, Exhaustive: 10
Step 3:
Greedy: 1, Beam: 3, Exhaustive: 1000

Show answer

Correct answer

D
Step 1:
Greedy: 1, Beam: 3, Exhaustive: 10
Step 3:
Greedy: 1, Beam: 3, Exhaustive: 1000

Question 9 on its own page

Question 10

+2 marksOne or more correct options

Which of the following decoding strategies is/are inherently non-deterministic?

Select all that apply.

A
Beam
B
Greedy
C
Nucleus
D
Exhaustive

Show answer

Correct answer

C
Nucleus

Question 10 on its own page

Question 11

+2 marksOne or more correct options

Which of the following statements about multiple attention heads in a Transformer are true?

Select all that apply.

A
They allow learning diverse local/global relations.
B
They reduce memory usage compared to one head.
C
Increasing the number of heads while keeping dmodel fixed, decreases the dimensionality handled by each head.
D
All heads necessarily learn completely independent information about the input.

Show answer

Correct answers

A
They allow learning diverse local/global relations.
C
Increasing the number of heads while keeping dmodel fixed, decreases the dimensionality handled by each head.

Question 11 on its own page

Question 12

+4 marksOne or more correct options

Which of the following statements about the Transformer architecture are true?

Select all that apply.

A
Masking (look-ahead mask) is applied in every decoder self-attention layer, not just the first one.
B
Token embeddings are used in the encoder but not in the decoder.
C
Positional encoding is required only in the encoder, not in the decoder.
D
Token embeddings are used in both the encoder and the decoder.
E
Positional encodings are added to the input embeddings in both the encoder and the decoder.

Show answer

Correct answers

A
Masking (look-ahead mask) is applied in every decoder self-attention layer, not just the first one.
D
Token embeddings are used in both the encoder and the decoder.
E
Positional encodings are added to the input embeddings in both the encoder and the decoder.

Question 12 on its own page

Question 13

+4 marksOne or more correct options

A transformer decoder uses a vocabulary of size 5. At step 1, the predicted token probabilities are:

Token	Probability
A	0.40
B	0.30
C	0.20
D	0.05
E	0.05

Which statement(s) is/are correct?

Select all that apply.

A
Beam search with beam size 2 will keep either A or B as per probability whereas Top-2 sampling will keep both A and B.
B
Beam search with beam size 2 will keep both A and B whereas Top-2 sampling will keep either A or B.
C
Beam search with beam size 2 will randomly choose any two tokens, whereas Top-2 sampling will always pick the two least probable tokens.
D
Beam search with beam size 2 will always keep the top-2 most probable tokens, whereas Top-2 sampling will restrict choices to top-2 tokens and then sample one probabilistically.

Show answer

Correct answers

B
Beam search with beam size 2 will keep both A and B whereas Top-2 sampling will keep either A or B.
D
Beam search with beam size 2 will always keep the top-2 most probable tokens, whereas Top-2 sampling will restrict choices to top-2 tokens and then sample one probabilistically.

Question 13 on its own page

Question 14

+3 marksNumerical answer

Show answer

Correct answer: 1152

Question 14 on its own page

Question 15

+3 marksNumerical answer

A Transformer model processes a sequence of $T = 1000$ tokens. Within the self-attention layer, the attention scores are stored as 32-bit floating-point numbers (float32). We compare two configurations that differ in the number of attention heads, $h$ :

$\text{Model A: } h = 8 \qquad \text{Model B: } h = 16$

Using the given conversion factors (1 byte $=$ 8 bits, 1 MB $= 10^6$ bytes.), calculate the percentage increase in memory (in MB) required to store all attention scores when moving from Model A to Model B.

Show answer

Correct answer: 100 (accepted within ±1)

Question 15 on its own page

Question 16

+2 marksNumerical answer

Show answer

Correct answer: 32768

Question 16 on its own page

Question 17

+3 marksNumerical answer

You are analyzing a GPT-style decoder-only Transformer trained as a causal language model with the following setup:

Sequence length: $T = 1024$ tokens per input sequence.
Batch size: $B = 32$ .
Hidden dimension: $d_{\text{model}} = 768$ .
Number of attention heads: $h = 8$ .
$d_v = \frac{d_{\text{model}}}{h}$
Training uses next-token prediction with teacher forcing and a causal mask.
The input sequence does not include a beginning-of-sequence (BOS) token.

Answer the given sub-questions:

How many next-token prediction targets are generated during training a full batch?

Show answer

Correct answer: 32736

Question 17 on its own page

Question 18

+2 marksNumerical answer

You are analyzing a GPT-style decoder-only Transformer trained as a causal language model with the following setup:

Sequence length: $T = 1024$ tokens per input sequence.
Batch size: $B = 32$ .
Hidden dimension: $d_{\text{model}} = 768$ .
Number of attention heads: $h = 8$ .
$d_v = \frac{d_{\text{model}}}{h}$
Training uses next-token prediction with teacher forcing and a causal mask.
The input sequence does not include a beginning-of-sequence (BOS) token.

Answer the given sub-questions:

For one sequence, how many non-zero attention scores remain in the masked attention score matrix (per head)?

Show answer

Correct answer: 524800

Question 18 on its own page

Question 19

+2 marksNumerical answer

You are analyzing a GPT-style decoder-only Transformer trained as a causal language model with the following setup:

Sequence length: $T = 1024$ tokens per input sequence.
Batch size: $B = 32$ .
Hidden dimension: $d_{\text{model}} = 768$ .
Number of attention heads: $h = 8$ .
$d_v = \frac{d_{\text{model}}}{h}$
Training uses next-token prediction with teacher forcing and a causal mask.
The input sequence does not include a beginning-of-sequence (BOS) token.

Answer the given sub-questions:

Show answer

Correct answer: 589824

Question 19 on its own page

Question 20

+2 marksOne or more correct options

You are analyzing a GPT-style decoder-only Transformer trained as a causal language model with the following setup:

Sequence length: $T = 1024$ tokens per input sequence.
Batch size: $B = 32$ .
Hidden dimension: $d_{\text{model}} = 768$ .
Number of attention heads: $h = 8$ .
$d_v = \frac{d_{\text{model}}}{h}$
Training uses next-token prediction with teacher forcing and a causal mask.
The input sequence does not include a beginning-of-sequence (BOS) token.

Answer the given sub-questions:

During autoregressive inference with GPT:

Select all that apply.

A
The model can generate all tokens in parallel.
B
Each new token is generated conditioned on previously generated tokens.
C
Teacher forcing is used to stabilize outputs.
D
Beam search or other decoding strategies are commonly used to decide the next token.

Show answer

Correct answers

B
Each new token is generated conditioned on previously generated tokens.
D
Beam search or other decoding strategies are commonly used to decide the next token.

Question 20 on its own page