Quiz Space

Deep Learning Practice · Quiz 1 · 26 Oct 2025 · September 2025 term

Question 14: A single transformer block in a GPT-style model has the …

Question 14

+4 marksNumerical answer

A single transformer block in a GPT-style model has the following configuration: embedding dimension (d model) = 1024, num attention heads = 16. Each attention head has a dimension of d model /num attention heads. Calculate the total number of parameters (weights and biases) for the self attention mechanism (specifically the Q, K, V, and Output projection layers) within this single block. Report the answer in millions, rounded to one decimal place.(in M)

Show answer

Correct answer: 4.2

See the whole paperTake it as a mock test

Question 14 of 15 in the IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 26 Oct 2025, in the September 2025 term (IIT M DEGREE AN EXAM QDB2 26 Oct 2025). It carries 4 marks.

More questions from this paper

  1. Q1Which of the following statements best describes the primary advantage of the SentencePiece tokenizer compared to a sta…
  2. Q2When performing full fine-tuning of a large language model (e.g., 10B parameters) using Adam, which component consumes …
  3. Q3A company wants to align its chatbot with values of being helpful, harmless, and honest. Human labelers provide ideal r…
  4. Q4Figure question
  5. Q5After running the following Hugging Face datasets code snippet, what will be the features of the dataset?
  6. Q6What is a key implication of using a Parameter-Efficient Fine-Tuning (PEFT) method like LoRA when adapting a large lang…
  7. Q7The WordPiece tokenization algorithm, unlike BPE, does not merge the pair with the highest frequency. Instead, it merge…
  8. Q8Which of the following statements accurately describes the Causal Language Modeling (CLM) objective used to pre-train m…
  9. Q9A research lab has access to a powerful 175B parameter language model. They need to adapt it for a highly specialized l…
  10. Q10A team is fine-tuning a 7B parameter model on a single GPU with 24GB of memory. They are using the Adam optimizer (whic…
  11. Q11The evolution of NLP models shows a distinct shift from task-specific architectures to a ”pre-train, finetune” paradigm…
  12. Q12The three main families of Transformer-based models are Encoder-only (e.g., BERT), Decoder-only (e.g.,GPT), and Encoder…
  13. Q13Figure question
  14. Q15You are given a GPT-style model with a vocabulary size of 50,257, a context length of 1024, and an embedding dimension …