LLM Quiz 2 PYQ Jan 2024: 24 Mar 2024 · IITM BS

Question 1

+3 marksNumerical answer

Three teams, namely A, B, and C, decided to use a GPT model with two decoder layers with the following configuration

length of context window ( $T$ ) $= 1024$
number of heads $n_h = 8$
$dmodel = 512$
$dff = 4 * dmodel$
$dq = dk = dv = \frac{dmodel}{n_h}$
The weights of the embedding layer and the output layer are shared (tied)

They train the model on the pre-training dataset containing one million words after applying normalization and pre-tokenization( using white space as a delimiter). Moreover, the dataset has no duplicate sentences and no upper-case letters or words. Suppose the team prefers to use the BPE tokenizer to build the vocabulary and subsequently use that to train the model. The base vocabulary contains lowercase alphabets (a to z), digits (0 to 9) and special tokens [unk],[go] and [end].

Team $A$ uses base vocabulary
Team $B$ takes the vocabulary from team $A$ and does 500 merges
Team $C$ takes the vocabulary from the team $B$ and does additional 500 merges

Based on the above data answer the given subquestions.

What is the size of the vocabulary built by the Team C?

Show answer

Correct answer: 1039

Question 1 on its own page

Question 2

+4 marksOne or more correct options

Three teams, namely A, B, and C, decided to use a GPT model with two decoder layers with the following configuration

length of context window ( $T$ ) $= 1024$
number of heads $n_h = 8$
$dmodel = 512$
$dff = 4 * dmodel$
$dq = dk = dv = \frac{dmodel}{n_h}$
The weights of the embedding layer and the output layer are shared (tied)

They train the model on the pre-training dataset containing one million words after applying normalization and pre-tokenization( using white space as a delimiter). Moreover, the dataset has no duplicate sentences and no upper-case letters or words. Suppose the team prefers to use the BPE tokenizer to build the vocabulary and subsequently use that to train the model. The base vocabulary contains lowercase alphabets (a to z), digits (0 to 9) and special tokens [unk],[go] and [end].

Team $A$ uses base vocabulary
Team $B$ takes the vocabulary from team $A$ and does 500 merges
Team $C$ takes the vocabulary from the team $B$ and does additional 500 merges

Based on the above data answer the given subquestions.

Choose the correct statements about the number of parameters in the model (excluding the embedding and output layer parameters)

Select all that apply.

A
Team A model has less number of parameters than the team B
B
Team B model has less number of parameters than the team C
C
Team C model has more number of parameters than the team A
D
All models have the same number of parameters

Show answer

Correct answer

D
All models have the same number of parameters

Question 2 on its own page

Question 3

+4 marksOne or more correct options

Three teams, namely A, B, and C, decided to use a GPT model with two decoder layers with the following configuration

length of context window ( $T$ ) $= 1024$
number of heads $n_h = 8$
$dmodel = 512$
$dff = 4 * dmodel$
$dq = dk = dv = \frac{dmodel}{n_h}$
The weights of the embedding layer and the output layer are shared (tied)

They train the model on the pre-training dataset containing one million words after applying normalization and pre-tokenization( using white space as a delimiter). Moreover, the dataset has no duplicate sentences and no upper-case letters or words. Suppose the team prefers to use the BPE tokenizer to build the vocabulary and subsequently use that to train the model. The base vocabulary contains lowercase alphabets (a to z), digits (0 to 9) and special tokens [unk],[go] and [end].

Team $A$ uses base vocabulary
Team $B$ takes the vocabulary from team $A$ and does 500 merges
Team $C$ takes the vocabulary from the team $B$ and does additional 500 merges

Based on the above data answer the given subquestions.

Assume that the word “acrophobia” is not present in the vocabulary, then which of the following tokenizer(s) is(are) guaranteed to tokenize this word into sub-words units (i.e., it does not return [unk] token)

Select all that apply.

A
The tokenizer used by the team A
B
the tokenizer used by the team B
C
the tokenizer used by the team C
D
None of the given options

Show answer

Correct answers

A
The tokenizer used by the team A
B
the tokenizer used by the team B
C
the tokenizer used by the team C

Question 3 on its own page

Question 4

+3 marksOne correct option

Three teams, namely A, B, and C, decided to use a GPT model with two decoder layers with the following configuration

length of context window ( $T$ ) $= 1024$
number of heads $n_h = 8$
$dmodel = 512$
$dff = 4 * dmodel$
$dq = dk = dv = \frac{dmodel}{n_h}$
The weights of the embedding layer and the output layer are shared (tied)

They train the model on the pre-training dataset containing one million words after applying normalization and pre-tokenization( using white space as a delimiter). Moreover, the dataset has no duplicate sentences and no upper-case letters or words. Suppose the team prefers to use the BPE tokenizer to build the vocabulary and subsequently use that to train the model. The base vocabulary contains lowercase alphabets (a to z), digits (0 to 9) and special tokens [unk],[go] and [end].

Team $A$ uses base vocabulary
Team $B$ takes the vocabulary from team $A$ and does 500 merges
Team $C$ takes the vocabulary from the team $B$ and does additional 500 merges

Based on the above data answer the given subquestions.

Assume that team A and team B pre-trained the model for 1 million steps. Suppose they want to fine-tune the model for sentiment classification tasks (where the dataset contains sentences, digits and emojis). Choose the correct statement

A
Team A can use the tokenizer from Team B and fine-tune the model and vice versa
B
We can not interchange both teams’ tokenizers while fine-tuning.

Show answer

Correct answer

A
Team A can use the tokenizer from Team B and fine-tune the model and vice versa

Question 4 on its own page

Question 5

+4 marksOne or more correct options

Suppose we have two decoder-only models, namely Model A and Model B. Assume we have two datasets, one for unsupervised pre-training (Web-Text) and the other for supervised training (text summarization). Model A was trained directly on the training set of the supervised dataset. Model B was trained on the pre-training dataset using the CLM objective. Suppose we want to test the performance of both models on the test set of summarization task. Then choose the correct statements

Select all that apply.

A
Model A can directly be used on the test set
B
It is possible to use Model B directly on the test set without fine-tuning on the training set
C
Model B should be fine-tuned first on the training set and then use the test set.

Show answer

Correct answers

A
Model A can directly be used on the test set
B
It is possible to use Model B directly on the test set without fine-tuning on the training set

Question 5 on its own page

Question 6

+4 marksOne or more correct options

The strikeout words in the passage given below denote the words to be dropped from the original sentence. “At the heart ~~of science is~~ an essential balance between two seemingly ~~contradictory attitudes~~ — an openness to new ideas, no matter how bizarre or ~~counterintuitive they may be~~, and the most ruthless skeptical scrutiny of all ideas, old and new. This is ~~how deep truths are winnowed~~ from deep nonsense”

Which of the following represents the correct input sequence, with sentinel tokens, to the baseline model that uses the pre-training denoising objective? [z] represents the end of the sentinel token in a sentence. The characters inside the square brackets are the sentinel tokens

Select all that apply.

A
[v] of science is [w] contradictory attitudes [x] counterintuitive they may be [y] how deep truths are winnowed [z]
B
[a] of science is [b] contradictory attitudes [c] counterintuitive they may be [d] how deep truths are winnowed [z]
C
[v] of science is [w] contradictory attitudes [x] counterintuitive they may be [y] how deep truths are winnowed
D
[a] of science is [b] contradictory attitudes [c] counterintuitive they may be [d] how deep truths are winnowed
E
[w] contradictory attitudes [v] of science is [x] counterintuitive they may be [y] how deep truths are winnowed [z]
F
[v] of science is [v] contradictory attitudes [x] counterintuitive they may be [x] how deep truths are winnowed [y]

Show answer

Correct answers

C
[v] of science is [w] contradictory attitudes [x] counterintuitive they may be [y] how deep truths are winnowed
D
[a] of science is [b] contradictory attitudes [c] counterintuitive they may be [d] how deep truths are winnowed

Question 6 on its own page

Question 7

+3 marksOne or more correct options

Which of the following models uses a form of denoising objective for pre-training?

Select all that apply.

A
BERT
B
GPT-1 and GPT-2
C
BART
D
T5

Show answer

Correct answers

A
BERT
C
BART
D
T5

Question 7 on its own page

Question 8

+4 marksOne or more correct options

Consider training an encoder-decoder model (say, T5 baseline) on a pre-training dataset that contains 1 Billion tokens. Let the batch size be 64 × 128, that is 2¹³ tokens. Suppose the model is trained for 2¹⁵ steps by varying the size of the data (all other aspects are fixed)
A. Train the model only on the first one million tokens, referred to as Model A
B. Train the model on the entire dataset, referred to as Model B
Based on the above data answer the given subquestions.

Which of the following statements is (are) correct under this setup?

Select all that apply.

A
Training loss of Model A will be less than training loss of Model B
B
Training loss of Model B will be less than training loss of Model A
C
Test loss of Model B will likely be less than test loss of Model A
D
Test loss of Model A will likely be less than test loss of Model B

Show answer

Correct answers

A
Training loss of Model A will be less than training loss of Model B
C
Test loss of Model B will likely be less than test loss of Model A

Question 8 on its own page

Question 9

+4 marksOne or more correct options

Consider training an encoder-decoder model (say, T5 baseline) on a pre-training dataset that contains 1 Billion tokens. Let the batch size be 64 × 128, that is 2¹³ tokens. Suppose the model is trained for 2¹⁵ steps by varying the size of the data (all other aspects are fixed)
A. Train the model only on the first one million tokens, referred to as Model A
B. Train the model on the entire dataset, referred to as Model B
Based on the above data answer the given subquestions.

Which of the following modifications will improve the performance of both models?

Select all that apply.

A
Increasing the training steps helps Model B more than Model A in terms of generalization performance
B
Increasing the training steps helps both Model A and Model B in terms of generalization performance
C
None of the given modifications helps the model to improve their generalization performance

Show answer

Correct answer

A
Increasing the training steps helps Model B more than Model A in terms of generalization performance

Question 9 on its own page

Question 10

+5 marksOne or more correct options

Consider a summarization task. Assume the prefix is “Summarize the following:” and the input sentence is “There are a lot of videos on YouTube etc. that give the appearance of education, but if you look closely they are really just entertainment. This is very convenient for everyone involved: the people watching enjoy thinking they are learning (but they are just having fun)” and the model is supposed to summarize it.Then which of the following architectures could be used for this task

Select all that apply.

A
A decoder-only model with a causal mask
B
A decoder-only model with a prefix mask
C
An encoder-only model with a fully visible mask
D
Encoder-Decoder model with fully visible mask in the encoder and causal mask in the decoder

Show answer

Correct answers

A
A decoder-only model with a causal mask
B
A decoder-only model with a prefix mask
D
Encoder-Decoder model with fully visible mask in the encoder and causal mask in the decoder

Question 10 on its own page

Question 11

+4 marksOne or more correct options

Suppose we are given a Model A with N layers and the dataset with D tokens. Choose the correct statements according to the scaling law.

Select all that apply.

A
Increasing the model size requires a proportionate increase in the dataset size to reduce the test loss
B
Increasing the model size requires a proportionate increase in the dataset size to reduce the training loss
C
Increasing the model size requires us to double the dataset size to reduce the test loss
D
Increasing the model size requires us to double the dataset size to reduce the train loss

Show answer

Correct answer

A
Increasing the model size requires a proportionate increase in the dataset size to reduce the test loss

Question 11 on its own page

Question 12

+3 marksOne correct option

Which of the following components in the data pre-processing pipeline removes pages that contain bad words?

A
Language Identification
B
Exact Deduplication
C
Fuzzy Deduplication
D
ML classifiers for quality filtering
E
Simple heuristics to detect toxic contents

Show answer

Correct answer

E
Simple heuristics to detect toxic contents

Question 12 on its own pageAlso asked in Quiz 2 1 Dec 2024

Question 13

+3 marksOne correct option

Suppose a team wants to build a dataset for a low-resource language. They took all Wikipedia dumps for that language and found that the dataset size was insufficient. Then the statement that using years of Common Crawl Snapshots would be helpful to increase the size of the dataset is

A
TRUE
B
FALSE

Show answer

Correct answer

A
TRUE

Question 13 on its own page

Question 14

+2 marksOne correct option

The statement that the presence of significant duplicate contents in the pre-training dataset degrades the performance of Large Language Models is

A
TRUE
B
FALSE

Show answer

Correct answer

A
TRUE

Question 14 on its own page