Here is a set of training arguments used by the GPT-2 model that was pre-trained on a dataset that contains 10 billion tokens. The context length of the model is modified to 2048, the vocabulary size is 50,257 and the embedding dimension is 768. The length of all the samples in a batch is equal to the context length of the model. training_args = TrainingArguments( output_dir='out', evaluation_strategy="steps", eval_steps=500, num_train_epochs=1, per_device_train_batch_size=16, per_device_eval_batch_size=16, tf32=True, gradient_accumulation_steps=2, adam_beta1=0.9, adam_beta2=0.999, learning_rate=2e-5, weight_decay=0.01, logging_dir='logs', logging_strategy="steps", logging_steps = 500, save_steps=5000, save_total_limit=20, report_to='wandb', ) Based on the above data, answer the given subquestions. Enter the number of tokens (in millions) processed by the model after 1000 steps. Enter the answer to 2 decimal places. For example, if your answer is 123456789, then enter it as 123.45.

Deep Learning Practice · Quiz 1 · 27 Oct 2024 · September 2024 term

Question 13: Here is a set of training arguments used by the GPT-2 mo…

Question 13

+5 marksNumerical answer

Here is a set of training arguments used by the GPT-2 model that was pre-trained on a dataset that contains 10 billion tokens. The context length of the model is modified to 2048, the vocabulary size is 50,257 and the embedding dimension is 768. The length of all the samples in a batch is equal to the context length of the model.

python

1training_args = TrainingArguments( output_dir='out',
2                              evaluation_strategy="steps",
3                              eval_steps=500,
4                              num_train_epochs=1,
5                              per_device_train_batch_size=16,
6                              per_device_eval_batch_size=16,
7                              tf32=True,
8                              gradient_accumulation_steps=2,
9                              adam_beta1=0.9,
10                              adam_beta2=0.999,
11                              learning_rate=2e-5,
12                              weight_decay=0.01,
13                              logging_dir='logs',
14                              logging_strategy="steps",
15                              logging_steps = 500,
16                              save_steps=5000,
17                              save_total_limit=20,
18                              report_to='wandb',
19                            )

Based on the above data, answer the given subquestions.

Enter the number of tokens (in millions) processed by the model after 1000 steps. Enter the answer to 2 decimal places. For example, if your answer is 123456789, then enter it as 123.45.

Show answer

Correct answer: 65.5 (accepted within ±0.2)

See the whole paper Take it as a mock test

Question 13 of 15 in the IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 27 Oct 2024, in the September 2024 term (IIT M DEGREE AN EXAM QDB2 27 Oct 2024). It carries 5 marks.

Question 13: Here is a set of training arguments used by the GPT-2 mo…

Question 13

More questions from this paper