Question 3: Three teams, namely A, B, and C, decided to use a GPT mod…

Three teams, namely A, B, and C, decided to use a GPT model with two decoder layers with the following configuration

• length of context window ($T$) $= 1024$
• number of heads $n_h = 8$
• $dmodel = 512$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$
• The weights of the embedding layer and the output layer are shared (tied)

They train the model on the pre-training dataset containing one million words after applying normalization and pre-tokenization( using white space as a delimiter). Moreover, the dataset has no duplicate sentences and no upper-case letters or words. Suppose the team prefers to use the BPE tokenizer to build the vocabulary and subsequently use that to train the model. The base vocabulary contains lowercase alphabets (a to z), digits (0 to 9) and special tokens [unk],[go] and [end].

• Team $A$ uses base vocabulary
• Team $B$ takes the vocabulary from team $A$ and does 500 merges
• Team $C$ takes the vocabulary from the team $B$ and does additional 500 merges

Based on the above data answer the given subquestions.

Assume that the word “acrophobia” is not present in the vocabulary, then which of the following tokenizer(s) is(are) guaranteed to tokenize this word into sub-words units (i.e., it does not return [unk] token)