Quiz Space

Large Language Models · End Term · 10 May 2026 · January 2026 term

Question 4: A transformer layer with absolute positional encoding and…

Question 4

+2 marksOne correct option

A transformer layer with absolute positional encoding and another with ALiBi are compared on sequences extended from length 128 to 512, without retraining.
Which numerical behavior is most likely for ALiBi in this extrapolation scenario?

  1. A

    Attention scores remain numerically well-shaped because ALiBi biases scale linearly with distance, not with a fixed maximum length.

  2. B

    The model cannot compute positional information beyond 128 because ALiBi stores one embedding per absolute position up to 128 only.

  3. C

    ALiBi attention becomes undefined for positions beyond 128 because its biases require precomputed lookup tables that stop at 128.

  4. D

    Attention scores collapse to near zero for distant positions because ALiBi uses fixed sinusoidal patterns tied to the original length 128.

Show answer

Correct answer

  • A

    Attention scores remain numerically well-shaped because ALiBi biases scale linearly with distance, not with a fixed maximum length.

See the whole paperTake it as a mock test

Question 4 of 20 in the IIT Madras BS Large Language Models (LLM) End Term paper sat on 10 May 2026, in the January 2026 term (Large Language Models 10 May 26). It carries 2 marks.

More questions from this paper

  1. Q1In the standard Transformer Decoder, the Multi-Head Attention layer is "Masked". What is the specific purpose of this m…
  2. Q2Figure question
  3. Q3Figure question
  4. Q5Based on the BERT pre-training phase, select all correct constraints and properties regarding the input data.
  5. Q6Select the statements that correctly distinguish fine-tuning from zero-shot prompting for GPT-style models.
  6. Q7Figure question
  7. Q8Figure question
  8. Q9Figure question
  9. Q10Suppose a language model outputs the following logits for a vocabulary of size 4:
  10. Q11Figure question
  11. Q12Adapter Layers are a parameter-efficient fine-tuning strategy. Where are these layers typically inserted within a Trans…
  12. Q13Why do standard LLM architectures include explicit Positional Encodings (like RoPE or Sinusoidal) despite the causal ma…
  13. Q14Figure question
  14. Q15Consider Multi-Query Attention (MQA). How does its architecture differ from standard Multi-Head Attention (MHA) to impr…
  15. Q16Figure question
  16. Q17Figure question
  17. Q18Which of the following statements about fine-tuning strategies such as gradual unfreezing and multi-task learning in th…
  18. Q19Select all correct statements regarding Local Attention Variants (like Sparse, Dilated, or Windowed Attention).
  19. Q20Which of the following techniques are used in Flash Attention to speed up attention computation and save memory?