Quiz Space

Deep Learning Practice · Quiz 1 · 23 Feb 2025 · January 2025 term

Question 16: Below is a snippet for loading the GPT-2 model configura…

Question 16

+3 marksOne correct option

Below is a snippet for loading the GPT-2 model configuration:

python
from transformers import GPT2Config


config = GPT2Config.from_pretrained("gpt2-medium")
print(config)

the output of the above is

text
GPT2Config {
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 1024,
  "n_head": 16,
  "n_inner": null,
  "n_layer": 24,
  "n_positions": 1024,
  "n_special": 0,
  "predict_special_tokens": true,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.1,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50
    }
   },
   "transformers_version": "4.47.1",
   "use_cache": true,
   "vocab_size": 50257
 }

Based on the above data, answer the given subquestions.

What does the n_ctx parameter in the GPT-2 configuration represent?

  1. A

    The number of attention heads in the model.

  2. B

    The maximum length of the input sequence in tokens.

  3. C

    The embedding size of each token.

  4. D

    The total number of parameters in the model.

Show answer

Correct answer

  • B

    The maximum length of the input sequence in tokens.

See the whole paperTake it as a mock test

Question 16 of 16 in the IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 23 Feb 2025, in the January 2025 term (IIT M DEGREE AN EXAM QDB2 23 Feb 2025). It carries 3 marks.

More questions from this paper

  1. Q1Figure question
  2. Q2Consider the following code where a tokenizer is created using the BPE model: Which of the following statements is corr…
  3. Q3Consider the following code snippet: What will likely be printed as the output?
  4. Q4Given the following code snippet: Which of the following options correctly adds LoRA to the base_model? (Whichever opti…
  5. Q5Given the following code for setting up the trainer: Which of the following lines correctly starts the fine-tuning proc…
  6. Q6What would happen if you set the target_modules parameter in the LoraConfig to an empty list []?
  7. Q7Consider the following code where the tokenizer is trained using two different vocabulary sizes: 5K and 50K. The functi…
  8. Q8What type of language modeling objective is used during the pretraining of GPT-2?
  9. Q9Which of the following is/are gradient based (fine tuning) methods?
  10. Q10Figure question
  11. Q11Select all the correct statements.
  12. Q12You are working with two datasets and trying to combine them using the following code: Assume the following: The ds1 da…
  13. Q13The original train split contains 25,000 samples, and 60% of the samples have a text length greater than 200. If the da…
  14. Q14Below is a snippet for loading the GPT-2 model configuration: the output of the above is Based on the above data, answe…
  15. Q15Below is a snippet for loading the GPT-2 model configuration: the output of the above is Based on the above data, answe…