Quiz Space

Deep Learning Practice · Quiz 1 · 23 Feb 2025 · January 2025 term

Question 15: Below is a snippet for loading the GPT-2 model configura…

Question 15

+4 marksNumerical answer

Below is a snippet for loading the GPT-2 model configuration:

python
from transformers import GPT2Config


config = GPT2Config.from_pretrained("gpt2-medium")
print(config)

the output of the above is

text
GPT2Config {
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 1024,
  "n_head": 16,
  "n_inner": null,
  "n_layer": 24,
  "n_positions": 1024,
  "n_special": 0,
  "predict_special_tokens": true,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.1,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50
    }
   },
   "transformers_version": "4.47.1",
   "use_cache": true,
   "vocab_size": 50257
 }

Based on the above data, answer the given subquestions.

Calculate the number of embedding parameters in the model. Enter your answer in millions, rounded to two decimal places.

Show answer

Correct answer: 51.5 (accepted within ±0.5)

See the whole paperTake it as a mock test

Question 15 of 16 in the IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 23 Feb 2025, in the January 2025 term (IIT M DEGREE AN EXAM QDB2 23 Feb 2025). It carries 4 marks.

More questions from this paper

  1. Q1Figure question
  2. Q2Consider the following code where a tokenizer is created using the BPE model: Which of the following statements is corr…
  3. Q3Consider the following code snippet: What will likely be printed as the output?
  4. Q4Given the following code snippet: Which of the following options correctly adds LoRA to the base_model? (Whichever opti…
  5. Q5Given the following code for setting up the trainer: Which of the following lines correctly starts the fine-tuning proc…
  6. Q6What would happen if you set the target_modules parameter in the LoraConfig to an empty list []?
  7. Q7Consider the following code where the tokenizer is trained using two different vocabulary sizes: 5K and 50K. The functi…
  8. Q8What type of language modeling objective is used during the pretraining of GPT-2?
  9. Q9Which of the following is/are gradient based (fine tuning) methods?
  10. Q10Figure question
  11. Q11Select all the correct statements.
  12. Q12You are working with two datasets and trying to combine them using the following code: Assume the following: The ds1 da…
  13. Q13The original train split contains 25,000 samples, and 60% of the samples have a text length greater than 200. If the da…
  14. Q14Below is a snippet for loading the GPT-2 model configuration: the output of the above is Based on the above data, answe…
  15. Q16Below is a snippet for loading the GPT-2 model configuration: the output of the above is Based on the above data, answe…