The IIT Madras BS Large Language Models (LLM) End Term paper sat on 10 May 2026, in the January 2026 term: 20 questions for 50 marks in 180 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.
In the standard Transformer Decoder, the Multi-Head Attention layer is "Masked". What is the specific purpose of this mask during training?
To filter out padding tokens to save computation.
To prevent the model from attending to the [CLS] and [SEP] special tokens.
To force the model to focus on the Encoder output rather than the Decoder input.
Correct answer
Correct answer
It has a limited receptive field and cannot capture long-range dependencies directly.
It requires more memory than full attention.
It cannot be parallelized on GPUs.
It causes the gradients to explode.
Correct answer
It has a limited receptive field and cannot capture long-range dependencies directly.
A transformer layer with absolute positional encoding and another with ALiBi are compared on sequences extended from length 128 to 512, without retraining.
Which numerical behavior is most likely for ALiBi in this extrapolation scenario?
Attention scores remain numerically well-shaped because ALiBi biases scale linearly with distance, not with a fixed maximum length.
The model cannot compute positional information beyond 128 because ALiBi stores one embedding per absolute position up to 128 only.
ALiBi attention becomes undefined for positions beyond 128 because its biases require precomputed lookup tables that stop at 128.
Attention scores collapse to near zero for distant positions because ALiBi uses fixed sinusoidal patterns tied to the original length 128.
Correct answer
Attention scores remain numerically well-shaped because ALiBi biases scale linearly with distance, not with a fixed maximum length.
Based on the BERT pre-training phase, select all correct constraints and properties regarding the input data.
The maximum context length (sequence length) supported is 512 tokens.
The input text must be contiguous (long-range contiguous text), meaning sentences are not shuffled randomly.
The vocabulary is built using BPE encoding.
Correct answers
The maximum context length (sequence length) supported is 512 tokens.
The input text must be contiguous (long-range contiguous text), meaning sentences are not shuffled randomly.
Select the statements that correctly distinguish fine-tuning from zero-shot prompting for GPT-style models.
In Fine-Tuning, the weights of the pre-trained model (and often a new specific head) are updated using backpropagation on a labeled dataset.
In Zero-Shot Prompting, the model weights are frozen (not updated); the task is formulated purely through the design of the input text (prompt).
Zero-Shot Prompting generally achieves higher accuracy than Fine-Tuning when abundant labeled data is available.
Fine-Tuning typically requires a task-specific output layer (like a linear classifier), whereas Zero-Shot Prompting relies on the model generating text directly.
Correct answers
In Fine-Tuning, the weights of the pre-trained model (and often a new specific head) are updated using backpropagation on a labeled dataset.
In Zero-Shot Prompting, the model weights are frozen (not updated); the task is formulated purely through the design of the input text (prompt).
Fine-Tuning typically requires a task-specific output layer (like a linear classifier), whereas Zero-Shot Prompting relies on the model generating text directly.
Correct answer: 3
Correct answer: 1048 (accepted within ±2)
Correct answer: 4
Suppose a language model outputs the following logits for a vocabulary of size 4:
Correct answer: 0.46 (accepted within ±0.03)
Correct answer: 640000
Adapter Layers are a parameter-efficient fine-tuning strategy. Where are these layers typically inserted within a Transformer block?
Before the Self-Attention mechanism only.
After the Feed-Forward Network (FFN) block and/or after the Self-Attention block.
Replacing the Layer Normalization parameters.
In parallel to the main residual connection.
Correct answer
After the Feed-Forward Network (FFN) block and/or after the Self-Attention block.
Why do standard LLM architectures include explicit Positional Encodings (like RoPE or Sinusoidal) despite the causal mask providing some implicit order?
To ensure that the model can only attend to future tokens.
To provide a strong inductive bias that allows the model to learn structural and distance-based relationships more efficiently than relying on masking alone.
To fix a mathematical error in the original Dot-Product Attention formula.
Correct answer
To provide a strong inductive bias that allows the model to learn structural and distance-based relationships more efficiently than relying on masking alone.
1 Key/Value head (same as MQA).
Correct answer
Consider Multi-Query Attention (MQA). How does its architecture differ from standard Multi-Head Attention (MHA) to improve inference efficiency?
It uses multiple query heads but only a single attention head.
It maintains multiple Query heads but shares a single Key and Value head across all Query heads.
It quantizes the Query vectors to 4 bits.
It removes the Value vectors entirely and uses Keys as Values.
Correct answer
It maintains multiple Query heads but shares a single Key and Value head across all Query heads.
Correct answer
Correct answer
Which of the following statements about fine-tuning strategies such as gradual unfreezing and multi-task learning in the context of the T5 model are correct?
In gradual unfreezing, layers are progressively unfrozen starting from the task-specific (top) layers toward the input (bottom) layers.
Multi-task learning in T5 typically improves performance uniformly across all tasks.
Multi-task pre-training (as used in T5) followed by task-specific fine-tuning often performs better than relying on multi-task learning alone.
Gradual unfreezing can help mitigate catastrophic forgetting of pre-trained representations during T5 fine-tuning.
Correct answers
In gradual unfreezing, layers are progressively unfrozen starting from the task-specific (top) layers toward the input (bottom) layers.
Multi-task learning in T5 typically improves performance uniformly across all tasks.
Gradual unfreezing can help mitigate catastrophic forgetting of pre-trained representations during T5 fine-tuning.
Select all correct statements regarding Local Attention Variants (like Sparse, Dilated, or Windowed Attention).
Many local attention variants reduce complexity toward linear with sequence length.
They always outperform full attention on all tasks, regardless of sequence length.
Stacking multiple layers of local attention increases the effective receptive field, similar to dilated convolutions in CNNs.
They can be combined with global tokens (e.g., in BigBird) to ensure some tokens can attend to the entire sequence.
Correct answers
Many local attention variants reduce complexity toward linear with sequence length.
Stacking multiple layers of local attention increases the effective receptive field, similar to dilated convolutions in CNNs.
They can be combined with global tokens (e.g., in BigBird) to ensure some tokens can attend to the entire sequence.
Which of the following techniques are used in Flash Attention to speed up attention computation and save memory?
Tiling: Breaking the large matrices into smaller blocks that fit into the fast SRAM.
Recomputation: Recomputing the attention matrix during the backward pass instead of storing it in HBM.
Approximation: Using a low-rank approximation of the attention matrix to avoid exact calculation.
Kernel Fusion: Fusing multiple operations (MatMul, Mask, Softmax, Dropout) into a single kernel to reduce memory reads/writes.
Correct answers
Tiling: Breaking the large matrices into smaller blocks that fit into the fast SRAM.
Recomputation: Recomputing the attention matrix during the backward pass instead of storing it in HBM.
Kernel Fusion: Fusing multiple operations (MatMul, Mask, Softmax, Dropout) into a single kernel to reduce memory reads/writes.
