Question 14
Consider a GPT model used for Causal language modelling. We feed the input sentence “This is a cool idea” to the model by appending special staring [BOS] and ending [EOS] tokens ( that is, “[BOS] This is a cool idea [EOS]”). Assume the context length of the model is 7. The attention matrix computed in one of the attention layers is given below
| 1 | 0 | 0 | 0 | 0 | 0 | 0 |
|---|---|---|---|---|---|---|
| 0.5 | 0.5 | 0 | 0 | 0 | 0 | 0 |
| 0.3 | 0.2 | 0.5 | 0 | 0 | 0 | 0 |
| 0.1 | 0.1 | 0.8 | 0 | 0 | 0 | 0 |
| 0 | 0.15 | 0 | 0.6 | 0.25 | 0 | 0 |
| 0.25 | 0.35 | 0 | 0.2 | 0.1 | 0.1 | 0 |
| 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.4 | 0.1 |
Table 1: Attention score
Based on the above data, answer the given subquestions.
Assume the time step starts from t = 0 and ends at t = 6. Suppose the model is at time step t = 4, what is the attention value assigned for the word “is”?