Question 13
Consider a GPT model used for Causal language modelling. We feed the input sentence “This is a cool idea” to the model by appending special staring [BOS] and ending [EOS] tokens ( that is, “[BOS] This is a cool idea [EOS]”). Assume the context length of the model is 7. The attention matrix computed in one of the attention layers is given below
| 1 | 0 | 0 | 0 | 0 | 0 | 0 |
|---|---|---|---|---|---|---|
| 0.5 | 0.5 | 0 | 0 | 0 | 0 | 0 |
| 0.3 | 0.2 | 0.5 | 0 | 0 | 0 | 0 |
| 0.1 | 0.1 | 0.8 | 0 | 0 | 0 | 0 |
| 0 | 0.15 | 0 | 0.6 | 0.25 | 0 | 0 |
| 0.25 | 0.35 | 0 | 0.2 | 0.1 | 0.1 | 0 |
| 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.4 | 0.1 |
Table 1: Attention score
Based on the above data, answer the given subquestions.
The attention score matrix given in Table 1 is appropriate for the causal language modelling task
True
False
Insufficient information