Opening the paper…
Large Language Models, End Term
Given the input string:
moonlight
And the following vocabulary of subword tokens with their corresponding log-probabilities:
| Subword | Log-Probability (base ) |
|---|---|
moon | -0.4 |
light | -1.0 |
moonlight | -1.8 |
moo | -0.3 |
nlight | -0.6 |
n | -2.0 |
li | -0.5 |
ght | -0.5 |
Using the Viterbi algorithm (as used in the SentencePiece tokenizer), determine the most probable tokenization of the input string. The probability of a tokenized sequence is the sum of the log-probabilities of the selected subwords. Only subwords from the vocabulary may be used.
Given the input string: `moonlight` And the following vocabulary of subword tokens with their corresponding log-probabilities: | Subword | Log-Probability (base $e$) | |---|---| | `moon` | -0.4 | | `light` | -1.0 | | `moonlight` | -1.8 | | `moo` | -0.3 | | `nlight` | -0.6 | | `n` | -2.0 | | `li` | -0.5 | | `ght` | -0.5 | Using the Viterbi algorithm (as used in the SentencePiece tokenizer), determine the most probable tokenization of the input string. The probability of a tokenized sequence is the sum of the log-probabilities of the selected subwords. Only subwords from the vocabulary may be used. Four 6x6 attention mask grids (a)-(d) with colored cells, to be matched with attention variants Which of the following statements about subword tokenizers (e.g., Byte Pair Encoding, SentencePiece) is **true**?