Question 9
Consider training an encoder-decoder model (say, T5 baseline) on a pre-training dataset that contains 1 Billion tokens. Let the batch size be 64 × 128, that is 2¹³ tokens. Suppose the model is trained for 2¹⁵ steps by varying the size of the data (all other aspects are fixed)
A. Train the model only on the first one million tokens, referred to as Model A
B. Train the model on the entire dataset, referred to as Model B
Based on the above data answer the given subquestions.
Which of the following modifications will improve the performance of both models?
Increasing the training steps helps Model B more than Model A in terms of generalization performance
Increasing the training steps helps both Model A and Model B in terms of generalization performance
None of the given modifications helps the model to improve their generalization performance