Opening the paper…
Large Language Models, End Term
Given the input string:
sunshine
And the following vocabulary of subword tokens with their corresponding log-probabilities:
| Subword | Log-Probability (base ) |
|---|---|
sun | -0.5 |
shine | -0.7 |
sunshine | -1.6 |
su | -0.3 |
nshine | -0.4 |
n | -1.8 |
shi | -0.6 |
ne | -0.5 |
Using the Viterbi algorithm (as used in the SentencePiece tokenizer), determine the most probable tokenization of the input string. The probability of a tokenized sequence is the sum of the log-probabilities of the selected subwords. Only subwords from the vocabulary may be used.
Given the input string: `sunshine` And the following vocabulary of subword tokens with their corresponding log-probabilities: | Subword | Log-Probability (base $e$) | |---|---| | `sun` | -0.5 | | `shine` | -0.7 | | `sunshine` | -1.6 | | `su` | -0.3 | | `nshine` | -0.4 | | `n` | -1.8 | | `shi` | -0.6 | | `ne` | -0.5 | Using the Viterbi algorithm (as used in the SentencePiece tokenizer), determine the most probable tokenization of the input string. The probability of a tokenized sequence is the sum of the log-probabilities of the selected subwords. Only subwords from the vocabulary may be used. A GPT model is trained using **causal language modeling**. During training, for a sequence of *T* = 4 tokens, which of the following correctly represents the **attention mask matrix** applied to the attention logits? Four 6x6 attention mask grids (a)-(d) with colored cells, to be matched with attention variants