Here is a set of training arguments used by the GPT-2 model that was pre-trained on a dataset that contains 10 billion tokens. The context length of the model is modified to 2048, the vocabulary size is 50,257 and the embedding dimension is 768. The length of all the samples in a batch is equal to the context length of the model. training_args = TrainingArguments( output_dir='out', evaluation_strategy="steps", eval_steps=500, num_train_epochs=1, per_device_train_batch_size=16, per_device_eval_batch_size=16, tf32=True, gradient_accumulation_steps=2, adam_beta1=0.9, adam_beta2=0.999, learning_rate=2e-5, weight_decay=0.01, logging_dir='logs', logging_strategy="steps", logging_steps = 500, save_steps=5000, save_total_limit=20, report_to='wandb', ) Based on the above data, answer the given subquestions. How many steps does it take to complete one epoch of training? Enter the answer in thousands (round down to an integer). For example, if your answer is 1234567.89, then enter it as 1234567.

Deep Learning Practice · Quiz 1 · 27 Oct 2024 · September 2024 term

Question 14: Here is a set of training arguments used by the GPT-2 mo…

Question 14

+3 marksNumerical answer

Here is a set of training arguments used by the GPT-2 model that was pre-trained on a dataset that contains 10 billion tokens. The context length of the model is modified to 2048, the vocabulary size is 50,257 and the embedding dimension is 768. The length of all the samples in a batch is equal to the context length of the model.

python

1training_args = TrainingArguments( output_dir='out',
2                              evaluation_strategy="steps",
3                              eval_steps=500,
4                              num_train_epochs=1,
5                              per_device_train_batch_size=16,
6                              per_device_eval_batch_size=16,
7                              tf32=True,
8                              gradient_accumulation_steps=2,
9                              adam_beta1=0.9,
10                              adam_beta2=0.999,
11                              learning_rate=2e-5,
12                              weight_decay=0.01,
13                              logging_dir='logs',
14                              logging_strategy="steps",
15                              logging_steps = 500,
16                              save_steps=5000,
17                              save_total_limit=20,
18                              report_to='wandb',
19                            )

Based on the above data, answer the given subquestions.

How many steps does it take to complete one epoch of training? Enter the answer in thousands (round down to an integer). For example, if your answer is 1234567.89, then enter it as 1234567.

Show answer

Correct answer: 152.5 (accepted within ±0.5)

See the whole paper Take it as a mock test

Question 14 of 15 in the IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 27 Oct 2024, in the September 2024 term (IIT M DEGREE AN EXAM QDB2 27 Oct 2024). It carries 3 marks.

Question 14: Here is a set of training arguments used by the GPT-2 mo…

Question 14

More questions from this paper