Question 12
Consider the following four pre-training scenarios for four different language models:
- Model A: Given “The cat sat on the ___”, predict “mat” using both left and right context.
- Model B: Given “The cat sat”, predict the next token “on”.
- Model C: Given corrupted input “The <X> sat on <Y> mat” (where spans are masked), generate “cat … the”.
- Model D: Given “<extra_id_0> cat sat <extra_id_1> mat”, generate “<extra_id_0> The <extra_id_1> on the”.
(Here, the symbols <X>, <Y>, and <extra_id_0>, <extra_id_1> represent different kinds of placeholder tokens used by some language models during pre-training.)
Match each scenario to the correct model:
A = BART, B = T5, C = BERT, D = GPT
A = T5, B = BERT, C = GPT, D = BART
A = BERT, B = GPT, C = BART, D = T5
A = BERT, B = BART, C = GPT, D = T5