Large Language Models End Term: 13 September 2026, Set S2 (May 2026 term)
The IIT Madras BS Large Language Models (LLM) End Term paper sat on 13 Sept 2026, in the May 2026 term, set S2: 20 questions for 50 marks in 180 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.
- 20
- 50
- 180 min
- 4
- 11
- 5
Show answer
Correct answer: 262144
Question 2
For T5-Base, the model dimension is 768 and FFN dimension is 3072. Ignore bias. Find the number of parameters in the FFN of one encoder layer.
Show answer
Correct answer: 4718592
Question 3
Show answer
Correct answer: 6400
Question 4
Calculate the KV Cache memory size in Megabytes for a single request (batch size = 1) with the following parameters:
• Number of layers: 24 • Embedding dimension (d_model): 1024 • Sequence length (n): 2048 tokens • Precision: 16-bit (2 bytes per element) • Assume standard Multi-Head Attention (MHA) where total KV dimension per layer is 2 x d_model (2 for K+V)
Show answer
Correct answer: 192
Question 5
In BERT's Next Sentence Prediction task, how are the negative (NotNext) examples created?
By taking the sentence that immediately precedes the current sentence.
By reversing the word order of the actual next sentence.
By picking a random sentence from the corpus (50% of the time).
By masking all verbs in the actual next sentence.
Show answer
Correct answer
By picking a random sentence from the corpus (50% of the time).
Question 6
What happens when a very large model is trained for many epochs on a relatively small dataset in the T5 scaling-law setting?
The performance continues to improve linearly with more training steps.
The performance degrades due to overfitting (memorization) of the training data.
The model automatically learns to augment the data, preventing overfitting.
The performance plateaus but does not degrade.
Show answer
Correct answer
The performance degrades due to overfitting (memorization) of the training data.
Question 7
PagedAttention is designed to address which specific inefficiency in Large Language Model (LLM) serving?
High computational latency of matrix multiplication.
Memory fragmentation and waste in the KV cache due to pre-allocating contiguous memory blocks for variable-length sequences.
Low bandwidth of the internet connection during API calls.
The difficulty of training models on long sequences.
Show answer
Correct answer
Memory fragmentation and waste in the KV cache due to pre-allocating contiguous memory blocks for variable-length sequences.
Question 8
Show answer
Correct answer
Question 9
In a standard Transformer layer utilizing Rotary Positional Embeddings (RoPE), at which specific stage of the forward pass is the rotary transformation applied?
Show answer
Correct answer
Question 10
The modification will not affect attention behavior significantly because softmax normalizes all scores, making additive biases irrelevant.
The modification will cause instability because the attention scores will become negative for nearby tokens and positive for distant tokens.
The modification is equivalent to scaling (Q) and (K) vectors differently and therefore does not fundamentally change the positional bias.
Show answer
Correct answer
Question 11
Comparing the architectures of BERT and GPT, select all the structural differences that are true.
BERT uses bidirectional self-attention, allowing tokens to attend to both left and right contexts.
GPT uses masked (causal) self-attention, allowing tokens to attend only to previous tokens.
BERT is an autoregressive model, while GPT is an autoencoding model.
During pre-training, BERT predicts masked tokens, whereas GPT predicts the next token in the sequence.
Show answer
Correct answers
BERT uses bidirectional self-attention, allowing tokens to attend to both left and right contexts.
GPT uses masked (causal) self-attention, allowing tokens to attend only to previous tokens.
During pre-training, BERT predicts masked tokens, whereas GPT predicts the next token in the sequence.
Question 12
Based on the architectural comparison experiments (Encoder-Decoder vs. Decoder-only vs. Encoder-only) discussed in the lectures:
Encoder-Decoder architectures generally perform best for sequence-to- sequence tasks like Translation and Summarization.
A Decoder-only model with a Prefix-LM objective can match the performance of an Encoder-Decoder model.
Sharing parameters between the encoder and decoder always improves performance regardless of task.
Encoder-only models (like BERT) are typically less suitable for generation tasks compared to Encoder-Decoder models.
Show answer
Correct answers
Encoder-Decoder architectures generally perform best for sequence-to- sequence tasks like Translation and Summarization.
A Decoder-only model with a Prefix-LM objective can match the performance of an Encoder-Decoder model.
Encoder-only models (like BERT) are typically less suitable for generation tasks compared to Encoder-Decoder models.
Question 13
Select all true statements regarding KV Caching during the autoregressive inference of LLMs.
It trades off increased memory usage for reduced computational latency.
It avoids recomputing the Key and Value vectors for tokens that have already been processed in previous steps.
It is essential during the training phase to speed up backpropagation.
The memory required for the KV cache grows linearly with the sequence length and batch size.
Show answer
Correct answers
It trades off increased memory usage for reduced computational latency.
It avoids recomputing the Key and Value vectors for tokens that have already been processed in previous steps.
The memory required for the KV cache grows linearly with the sequence length and batch size.
Question 14
Select all correct statements regarding Decoding Strategies in language generation.
Greedy search is deterministic and often leads to repetitive or degenerative text loops.
Show answer
Correct answers
Greedy search is deterministic and often leads to repetitive or degenerative text loops.
Question 15
In a feature-based approach using a pretrained transformer (like BERT) for a Multiple-Choice QA task, which of the following statements regarding the methodology are correct? (Select all that apply)
The weights of the pretrained transformer are updated via backpropagation during the training phase.
The input must be formatted as multiple pairs, such as [CLS] Question [SEP] Choice_n, creating a distinct representation for each candidate answer.
The final prediction is determined by passing the fixed output representations through a trainable linear layer and applying a Softmax function.
The model generates the correct answer string token-by-token using an autoregressive decoding strategy like Greedy Search.
Show answer
Correct answers
The input must be formatted as multiple pairs, such as [CLS] Question [SEP] Choice_n, creating a distinct representation for each candidate answer.
The final prediction is determined by passing the fixed output representations through a trainable linear layer and applying a Softmax function.
Question 16
A Transformer processes a sequence of length 5 using 6 identical self-attention layers. Count the total number of allowed attention links across all layers and Choose the option representing the correct pair
(A) BERT (B) GPT
BERT: 150, GPT: 90
BERT: 90, GPT: 150
BERT: 150, GPT: 150
BERT: 90, GPT: 90
BERT: 30, GPT: 36
BERT: 36, GPT: 30
Show answer
Correct answer
BERT: 150, GPT: 90
Question 17
Consider the architecture of BART. Which of the following descriptions best match its structural design?
A Decoder-only model similar to GPT, but with bidirectional attention in the first layer.
An Encoder-only model similar to BERT, but trained with a causal masking objective.
An encoder-decoder model where the encoder processes a corrupted input bidirectionally and the decoder autoregressively reconstructs the original sequence.
A dual-encoder model where one encoder processes the context and another processes the query.
Show answer
Correct answer
An encoder-decoder model where the encoder processes a corrupted input bidirectionally and the decoder autoregressively reconstructs the original sequence.
Question 18
Why is Flash Attention considered an "IO-aware" algorithm?
It reduces the number of parameters in the model to fit in GPU memory.
It compresses the input data using JPEG-like encoding before processing.
It optimizes the movement of data between the high-bandwidth memory (HBM) and the faster on-chip SRAM to avoid memory bandwidth bottlenecks.
It writes all intermediate attention matrices to the hard disk to save RAM.
Show answer
Correct answer
It optimizes the movement of data between the high-bandwidth memory (HBM) and the faster on-chip SRAM to avoid memory bandwidth bottlenecks.
Question 19
Show answer
Correct answer
Question 20
An engineer is designing a Transformer model capable of length extrapolation (handling sequences longer than those seen during training) while minimizing the number of trainable weights. They are evaluating Absolute Positional Encodings (APE), Relative Positional Encodings (RPE), Rotary Positional Encodings (RoPE), and Attention with Linear Biases (ALiBi).Which of the following statements correctly describes the parameterization and behavior of these techniques?
The engineer should choose APE or RPE to minimize weights, as both rely exclusively on fixed sinusoidal functions without requiring a lookup table.
RoPE and ALiBi are preferable for this use case because they utilize fixed mathematical transformations, whereas standard APE and RPE typically require learning position-specific embedding weights.
ALiBi is the only technique among the four that requires the model to learn a unique "slope" parameter for each attention head to determine the rate of penalty decay.
Show answer
Correct answer
RoPE and ALiBi are preferable for this use case because they utilize fixed mathematical transformations, whereas standard APE and RPE typically require learning position-specific embedding weights.