Question 8
Consider the following dictionary with the number of word occurrences in a corpus:
wo = {'simulation</w>': 4, 'suspicious</w>': 3, 'stringent</w>': 4,}Note: Identifier/special symbol </w> is already appended.
Suppose the process of building the vocabulary is stopped after 4 merges with BPE tokenization . The following is the list of first 4 merges in that order:
| Merge No. | Token 1 | Token 2 | Merged |
|---|---|---|---|
| 1 | ‘i’ | ‘o’ | ‘io’ |
| 2 | ‘u’ | ‘s’ | ‘us’ |
| 3 | ‘s’ | ‘i’ | ‘si’ |
| 4 | ‘si’ | ‘m’ | ‘sim’ |
Table 1: First 4 merges
Based on the above data, answer the given subquestions.
In how many tokens will the word ‘stereotypical</w>’ be tokenized? If this word can not be tokenized with the given vocabulary, then enter -1.
Note: Identifier/special symbol </w> is already appended.