Large Language Models End Term: 31 August 2025, Set QDB3 (May 2025 term)

Given the input string:

sunshine

And the following vocabulary of subword tokens with their corresponding log-probabilities:

Using the Viterbi algorithm (as used in the SentencePiece tokenizer), determine the most probable tokenization of the input string. The probability of a tokenized sequence is the sum of the log-probabilities of the selected subwords. Only subwords from the vocabulary may be used.

A GPT model is trained using causal language modeling. During training, for a sequence of T = 4 tokens, which of the following correctly represents the attention mask matrix applied to the attention logits?

Which of the following preprocessing steps are commonly used when preparing data for transformer models like BERT?

Which modifications are used in transformers to handle longer sequences efficiently?

In Transformer models that use relative position embeddings (such as in Transformer-XL or T5), clipping is often applied to the relative position indices. Which of the following statements about clipping in relative position embeddings are correct?

A vocabulary $\mathcal{V} = \{\text{tea}, \text{you}, \text{enjoy}, \text{often}\}$ is associated with the following embedding matrix:

$E = \begin{bmatrix} 0 & 1 \\ 1 & 1 \\ 1 & 0 \\ -1 & 1 \end{bmatrix}$

Words are indexed in the order shown above: “tea” (index 0), “you” (1), “enjoy” (2), and “often” (3). No positional encodings are used.

The parameters of the attention layer are:

$W_Q = \begin{bmatrix} 0.5 & 1 \\ 1 & 0.5 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 1 & 0.2 \\ 0.5 & 1 \end{bmatrix},$

$W_O = \begin{bmatrix} 0.5 & 0.5 \\ 1 & -1 \end{bmatrix}$

For the input sequence “you enjoy tea often”. The computed attention matrix $A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)$ is given as:

$A = \begin{bmatrix} 0.55 & 0.32 & 0.11 & 0.02 \\ 0.43 & 0.25 & 0.21 & 0.1 \\ 0.39 & 0.39 & 0.16 & 0.07 \\ 0.24 & 0.4 & 0.2 & 0.17 \end{bmatrix}$

Answer the given subquestions. If you believe the information is insufficient, enter -1 as your answer.

For the input “you enjoy tea often”, compute the final representation of the word “enjoy” after the attention layer (i.e., after applying WQ, WK, WV , attention weights, and WO). Enter the sum of the elements in the resulting vector.

A vocabulary $\mathcal{V} = \{\text{tea}, \text{you}, \text{enjoy}, \text{often}\}$ is associated with the following embedding matrix:

$E = \begin{bmatrix} 0 & 1 \\ 1 & 1 \\ 1 & 0 \\ -1 & 1 \end{bmatrix}$

Words are indexed in the order shown above: “tea” (index 0), “you” (1), “enjoy” (2), and “often” (3). No positional encodings are used.

The parameters of the attention layer are:

$W_Q = \begin{bmatrix} 0.5 & 1 \\ 1 & 0.5 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 1 & 0.2 \\ 0.5 & 1 \end{bmatrix},$

$W_O = \begin{bmatrix} 0.5 & 0.5 \\ 1 & -1 \end{bmatrix}$

For the input sequence “you enjoy tea often”. The computed attention matrix $A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)$ is given as:

$A = \begin{bmatrix} 0.55 & 0.32 & 0.11 & 0.02 \\ 0.43 & 0.25 & 0.21 & 0.1 \\ 0.39 & 0.39 & 0.16 & 0.07 \\ 0.24 & 0.4 & 0.2 & 0.17 \end{bmatrix}$

Answer the given subquestions. If you believe the information is insufficient, enter -1 as your answer.

Suppose the input sentence is “tea you enjoy often”. Using the same matrices and processing method, what is the attention score (i.e., softmax entry from A) for the query word “tea” attending to key word “often”?

Based on the above data, answer the given subquestions.

Choose the correct representation of π for the given mask image.

Based on the above data, answer the given subquestions.

How many permutations of π are possible for n = 5?

Based on the above data, answer the given subquestions.

What percentage of entries in the attention matrix are zero (i.e., the sparsity)?

Consider the short sentence:

“small models sometimes beat big ones”

A model processes this sequence using the naive relative positional embedding method. The model's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 6$. We use zero-based indexing for token positions:

$0: \texttt{small} \quad 1: \texttt{models} \quad 2: \texttt{sometimes} \quad 3: \texttt{beat} \quad 4: \texttt{big} \quad 5: \texttt{ones}$

Note: In this context, the relative position between two tokens is:

$\text{Relative position} = j - i$

(rows $=$ current token $i$, columns $=$ other token $j$).

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \\ 6 & 6 & 6 & 6 \end{bmatrix}.$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5}\right].$

Based on the above data, answer the given subquestions.

Consider the short sentence:

“small models sometimes beat big ones”

A model processes this sequence using the naive relative positional embedding method. The model's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 6$. We use zero-based indexing for token positions:

$0: \texttt{small} \quad 1: \texttt{models} \quad 2: \texttt{sometimes} \quad 3: \texttt{beat} \quad 4: \texttt{big} \quad 5: \texttt{ones}$

Note: In this context, the relative position between two tokens is:

$\text{Relative position} = j - i$

(rows $=$ current token $i$, columns $=$ other token $j$).

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \\ 6 & 6 & 6 & 6 \end{bmatrix}.$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5}\right].$

Based on the above data, answer the given subquestions.

Consider the short sentence:

“small models sometimes beat big ones”

A model processes this sequence using the naive relative positional embedding method. The model's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 6$. We use zero-based indexing for token positions:

$0: \texttt{small} \quad 1: \texttt{models} \quad 2: \texttt{sometimes} \quad 3: \texttt{beat} \quad 4: \texttt{big} \quad 5: \texttt{ones}$

Note: In this context, the relative position between two tokens is:

$\text{Relative position} = j - i$

(rows $=$ current token $i$, columns $=$ other token $j$).

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \\ 6 & 6 & 6 & 6 \end{bmatrix}.$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5}\right].$

Based on the above data, answer the given subquestions.

With a naive implementation, for each token embedding we combine (by addition) relative position embeddings. For token index $i \in \{0, \ldots, T-1\}$,

$\boldsymbol{h}_i = \boldsymbol{x}_i + \sum_{j=0}^{T-1} \boldsymbol{p}_{j-i}$

compute the final combined embedding $\boldsymbol{h}_i$ for the token "models" (i.e., $i = 1$). Submit the sum of all the elements of $\boldsymbol{h}_1$

Consider the short sentence:

“small models sometimes beat big ones”

A model processes this sequence using the naive relative positional embedding method. The model's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 6$. We use zero-based indexing for token positions:

$0: \texttt{small} \quad 1: \texttt{models} \quad 2: \texttt{sometimes} \quad 3: \texttt{beat} \quad 4: \texttt{big} \quad 5: \texttt{ones}$

Note: In this context, the relative position between two tokens is:

$\text{Relative position} = j - i$

(rows $=$ current token $i$, columns $=$ other token $j$).

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \\ 6 & 6 & 6 & 6 \end{bmatrix}.$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5}\right].$

Based on the above data, answer the given subquestions.

Consider the short sentence:

“small models sometimes beat big ones”

A model processes this sequence using the naive relative positional embedding method. The model's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 6$. We use zero-based indexing for token positions:

$0: \texttt{small} \quad 1: \texttt{models} \quad 2: \texttt{sometimes} \quad 3: \texttt{beat} \quad 4: \texttt{big} \quad 5: \texttt{ones}$

Note: In this context, the relative position between two tokens is:

$\text{Relative position} = j - i$

(rows $=$ current token $i$, columns $=$ other token $j$).

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \\ 6 & 6 & 6 & 6 \end{bmatrix}.$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5} \quad \frac{(j-i)}{5}\right].$

Based on the above data, answer the given subquestions.

Now consider the same sequence but assume the model uses the ALiBi (Attention with Linear Biases) method. The model has $H = 4$ heads and the per-head slope is defined as

$m_h = \frac{1}{2^h}, h = 0, 1, 2, 3,$

The pre-attention $e_{ij}$ for head $h$ are computed as:

$x_i W_Q x_j W_K \; + \; m_h \cdot \left[(j - i)\right]$

Assume $W_Q = I$ and $W_K = 2I$ where $I$ is an identity matrix . Answer $e_{ij}$ for the current token “big” and other token “small” for head $h = 2$. correct upto 1 digit after the decimal.