uiz Space

January 2026 term · Deep Learning Practice · BSDA5013

Deep Learning Practice Quiz 1: 15 March 2026 (January 2026 term)

The IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 15 Mar 2026, in the January 2026 term: 19 questions for 50 marks in 120 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.

Questions
19
Marks
50
Duration
120 min
MCQ
10
MSQ
6
Written
3
Take as mock testPractise with answers

Updated

Official paper: Deep Learning Practice 15 Mar 26 · No negative marking.

Question 1

+2 marksOne correct option

Consider subword tokenization methods such as Byte Pair Encoding (BPE) and WordPiece as employed in transformer-based language modeling pipelines. Which of the following statements correctly describes their fundamental properties?

  1. A

    They always split words at true morpheme boundaries (the smallest units of meaning, e.g., “un-” + “break” + “able”).

  2. B

    They learn merge rules from corpus statistics (frequency/likelihood), not token semantics

  3. C

    They eliminate (Out Of Vocabulary) OOV by ensuring every word is a single token

  4. D

    They generalize perfectly to out-of-domain text without increasing sequence length.

Show answer

Correct answer

  • B

    They learn merge rules from corpus statistics (frequency/likelihood), not token semantics

Question 1 on its own page

Question 2

+2 marksOne correct option

Consider two subword tokenizers trained on the same text corpus. Tokenizer A uses a vocabulary of 8,000 tokens, whereas Tokenizer B uses a vocabulary of 64,000 tokens. Which of the following statements most accurately characterizes the implications of these vocabulary sizes for training efficiency in transformer-based language models?

  1. A

    Larger vocabulary always reduces sequence length and total compute.

  2. B

    Smaller vocabulary always improves semantic alignment.

  3. C

    Larger vocabulary reduces sequence length but increases embedding parameters.

  4. D

    Vocabulary size has no effect once model is pretrained.

Show answer

Correct answer

  • C

    Larger vocabulary reduces sequence length but increases embedding parameters.

Question 2 on its own page

Question 3

+2 marksOne correct option

A subword tokenizer decomposes rare chemical entity names into a large number of fragments. During downstream fine-tuning, the model exhibits degraded performance on a chemical named entity recognition (NER) task. Which of the following represents the most principled corrective action?

  1. A

    Increase the dropout rate during fine-tuning.

  2. B

    Retrain the tokenizer using an in-domain (chemical) corpus.

  3. C

    Reduce the batch size during training.

  4. D

    Apply label smoothing to the loss function.

Show answer

Correct answer

  • B

    Retrain the tokenizer using an in-domain (chemical) corpus.

Question 3 on its own page

Question 4

+2 marksOne correct option
Figure from the original question paper
  1. A

    The output sequence will always contain exactly 12 non-padding tokens.

  2. B

    Special tokens may consume a portion of the 12-token length budget.

  3. C

    Word boundaries are preserved, since truncation occurs at whitespace boundaries.

  4. D

    Padding will never be applied when truncation is enabled.

Show answer

Correct answer

  • B

    Special tokens may consume a portion of the 12-token length budget.

Question 4 on its own page

Question 5

+2 marksOne correct option
Figure from the original question paper
  1. A

    This is supervised fine-tuning with noisy labels.

  2. B

    This is instruction tuning because labels are present.

  3. C

    This performs continual pretraining with MLM objective.

  4. D

    This trains a classifier head implicitly.

Show answer

Correct answer

  • C

    This performs continual pretraining with MLM objective.

Question 5 on its own page

Question 6

+2 marksOne correct option

In the standard Transformer architecture (e.g., GPT or BERT), which component exhibits a computational complexity that scales quadratically ( ) with respect to the input sequence length ?

  1. A

    The initial token and positional embedding lookup.

  2. B

    The element-wise addition in the Residual (Skip) connections.

  3. C

    The computation of the attention score matrix ( ).

  4. D

    The linear projections in the Position-wise Feed-Forward Network (FFN).

Show answer

Correct answer

  • C

    The computation of the attention score matrix ( ).

Question 6 on its own page

Question 7

+2 marksOne correct option

A model with 4B parameters (fp32) is being trained using the AdamW optimizer on a single 32GB GPU. Why is full fine-tuning impossible in this configuration?

  1. A

    The 4B parameters alone require 32GB of VRAM, leaving no room for the OS or CUDA kernels.

  2. B

    The optimizer states (32GB) plus the model weights (16GB) and gradients (16GB) exceed the 32GB VRAM limit.

  3. C

    The tokenizer is unable to map a context length of 2048 to a 32-bit integer space.

  4. D

    The fp32 precision requires 8 bytes per parameter, meaning the 4B model needs 64GB just to load.

Show answer

Correct answer

  • B

    The optimizer states (32GB) plus the model weights (16GB) and gradients (16GB) exceed the 32GB VRAM limit.

Question 7 on its own page

Question 8

+2 marksOne correct option

During the full fine-tuning of a Large Language Model, you aim to regularize the training process by penalizing the growth of weight magnitudes. This ensures the model does not overfit the fine-tuning data by making excessively large updates to the pretrained parameters. Which parameter is specifically designed to apply this penalty?

  1. A

    Gradient Accumulation.

  2. B

    Batch Size.

  3. C

    Checkpointing.

  4. D

    Weight Decay.

Show answer

Correct answer

  • D

    Weight Decay.

Question 8 on its own page

Question 9

+2 marksOne correct option

Why does full fine-tuning of large language models often require more memory than inference using the same model?

  1. A

    Inference uses larger batch sizes.

  2. B

    Inference stores optimizer states.

  3. C

    Training requires storing gradients and optimizer states.

  4. D

    Training uses longer input sequences.

Show answer

Correct answer

  • C

    Training requires storing gradients and optimizer states.

Question 9 on its own page

Question 10

+2 marksOne correct option

Which technique is most effective at reducing GPU memory usage during training without modifying the model architecture?

  1. A

    Gradient clipping.

  2. B

    Mixed-precision training.

  3. C

    Increasing learning rate warmup.

  4. D

    Saving checkpoints less frequently.

Show answer

Correct answer

  • B

    Mixed-precision training.

Question 10 on its own page

Question 11

+3 marksOne or more correct options

Consider the output of a Hugging Face tokenizer when applied to a batch of sequences with padding= "max_length" and return_tensors= "pt". Which of the following statements are correct? (Select ALL that apply)

Select all that apply.

  1. A

    The input_ids tensor contains the numerical indices mapped from the vocabulary.

  2. B

    The attention_mask contains 0s for padding tokens and 1s for real tokens to prevent the model from attending to padding.

  3. C

    The token_type_ids (if present) are used primarily to distinguish between uppercase and lowercase letters.

  4. D

    In models like BERT, the input_ids will typically begin with a special token index (e.g., [CLS]).

Show answer

Correct answers

  • A

    The input_ids tensor contains the numerical indices mapped from the vocabulary.

  • B

    The attention_mask contains 0s for padding tokens and 1s for real tokens to prevent the model from attending to padding.

  • D

    In models like BERT, the input_ids will typically begin with a special token index (e.g., [CLS]).

Question 11 on its own page

Question 12

+3 marksOne or more correct options
Figure from the original question paper

Select all that apply.

  1. A

    Effective batch size is larger than 1.

  2. B

    Reduced memory usage compared to FP32 training.

  3. C

    Gradient clipping limits large parameter updates.

  4. D

    Gradient accumulation reduces the number of optimizer states.

Show answer

Correct answers

  • A

    Effective batch size is larger than 1.

  • B

    Reduced memory usage compared to FP32 training.

  • C

    Gradient clipping limits large parameter updates.

Question 12 on its own page

Question 13

+3 marksOne or more correct options

Which of the following statements correctly distinguish different Transformer architectures and their typical training objectives? (Select ALL that apply)

Select all that apply.

  1. A

    Encoder-only models are commonly trained with Masked Language Modeling objectives.

  2. B

    Decoder-only models rely on causal masking and autoregressive inference.

  3. C

    Encoder-decoder models are well suited for sequence-to-sequence tasks.

  4. D

    All Transformer architectures use identical attention masks.

Show answer

Correct answers

  • A

    Encoder-only models are commonly trained with Masked Language Modeling objectives.

  • B

    Decoder-only models rely on causal masking and autoregressive inference.

  • C

    Encoder-decoder models are well suited for sequence-to-sequence tasks.

Question 13 on its own page

Question 14

+3 marksOne or more correct options
Figure from the original question paper

Select all that apply.

  1. A

    Task Conditioning: The model learns to follow natural language formatting (Instruction/Input/Answer) rather than just mapping a sequence to a single integer ID.

  2. B

    Open Vocabulary: The model's prediction head remains the size of its full vocabulary (e.g., 50k+ tokens), allowing it to generate any string as an answer instead of being restricted to fixed logits for classes.

  3. C

    Guaranteed Zero-Shot: This training format ensures the model will generalize perfectly to any unseen task prompt without further data.

  4. D

    Objective Shift: The training objective moves from minimizing cross-entropy loss over a discrete class index to minimizing next-token prediction loss over the sequence tokens.

Show answer

Correct answers

  • A

    Task Conditioning: The model learns to follow natural language formatting (Instruction/Input/Answer) rather than just mapping a sequence to a single integer ID.

  • B

    Open Vocabulary: The model's prediction head remains the size of its full vocabulary (e.g., 50k+ tokens), allowing it to generate any string as an answer instead of being restricted to fixed logits for classes.

  • D

    Objective Shift: The training objective moves from minimizing cross-entropy loss over a discrete class index to minimizing next-token prediction loss over the sequence tokens.

Question 14 on its own page

Question 15

+3 marksOne or more correct options

A model is trained using the Causal Language Modeling (CLM) objective with a standard cross-entropy loss. Which of the following statements correctly describe the training dynamics? (Select ALL that apply)

Select all that apply.

  1. A

    The loss at position depends only on tokens .

  2. B

    The attention mask is strictly upper triangular.

  3. C

    Future tokens contribute gradients to earlier positions.

  4. D

    The joint probability of the sequence is factorized autoregressively.

Show answer

Correct answers

  • A

    The loss at position depends only on tokens .

  • B

    The attention mask is strictly upper triangular.

  • D

    The joint probability of the sequence is factorized autoregressively.

Question 15 on its own page

Question 16

+3 marksOne or more correct options

When configuring TrainingArguments for a Transformer model, we include a learning rate warmup phase (e.g., warmup_steps=500). Which of the following statements correctly describe the purpose and behavior of this strategy? (Select ALL that apply)

Select all that apply.

  1. A

    It involves linearly increasing the learning rate from 0 (or a small value) to the target maximum during the initial phase.

  2. B

    It helps prevent "divergence'' or instability caused by large gradients when the model weights are far from their optimal state.

  3. C

    It significantly reduces the VRAM (memory) required to store optimizer states.

  4. D

    After the warmup phase, the learning rate typically follows a decay schedule (like linear or cosine) to ensure convergence.

Show answer

Correct answers

  • A

    It involves linearly increasing the learning rate from 0 (or a small value) to the target maximum during the initial phase.

  • B

    It helps prevent "divergence'' or instability caused by large gradients when the model weights are far from their optimal state.

  • D

    After the warmup phase, the learning rate typically follows a decay schedule (like linear or cosine) to ensure convergence.

Question 16 on its own page

Question 17

+4 marksWritten answer

A transformer-based language model has 1.2 billion parameters and is fine-tuned using AdamW. Assume parameters, gradients, and optimizer states (first and second moments) are all stored in 32-bit precision (4 bytes each). Ignoring activations and buffers, calculate the total GPU memory required (in GB) to store parameters, gradients, and optimizer states. (Assume 1 GB = bytes.)

Show answer

A written answer, not marked automatically.

Question 17 on its own page

Question 18

+4 marksWritten answer

A GPT-style transformer block has an embedding dimension of 768 and uses 12 attention heads. Consider a specific processing task with a very short sequence of only 3 tokens ( ).Compute the total number of weight parameters (ignore biases) in the self-attention module, specifically for the Query, Key, Value, and Output projection matrices. Report your answer in millions, rounded to one decimal place.

Show answer

A written answer, not marked automatically.

Question 18 on its own page

Question 19

+4 marksWritten answer

You are training a decoder-only language model with a vocabulary size of 32,000, maximum context length of 2048, and embedding dimension of 1024. The model uses learned positional embeddings. Calculate the total number of parameters in the embedding layer (token + positional embeddings). Report your answer in millions, rounded to one decimal place.

Show answer

A written answer, not marked automatically.

Question 19 on its own page