Large Language Models End Term: 31 August 2025, Set QDB1 (May 2025 term)

Given the input string:

moonlight

And the following vocabulary of subword tokens with their corresponding log-probabilities:

Using the Viterbi algorithm (as used in the SentencePiece tokenizer), determine the most probable tokenization of the input string. The probability of a tokenized sequence is the sum of the log-probabilities of the selected subwords. Only subwords from the vocabulary may be used.

Which of the following statements about subword tokenizers (e.g., Byte Pair Encoding, SentencePiece) is true?

A GPT-based model is fine-tuned for story generation. During evaluation, you test different decoding strategies on the same prompt:
1. Method A: Every run produces the exact same story, even when executed multiple times. 2. Method B: The story changes slightly in each run, but overall structure and key events remain similar.
3. Method C: The story is wildly different each time, sometimes even changing the genre. Based on the observations, select the most likely decoding strategies for each method:

Which of the following are valid design choices when building a transformer-based language model?

Which of the following statements about Top-k and Nucleus (Top-p) sampling are correct?

Which of the following preprocessing steps are commonly required when preparing data for transformer models like BERT?

Suppose a language model outputs the following logits for a vocabulary of size 4:

You apply Low Temperature Sampling with temperature $T = 2.0$. After applying the softmax on the temperature-scaled logits, what is the probability assigned to the token “cat”? (Give your answer up to 2 decimal places)

A vocabulary $\mathcal{V} = \{\text{tea}, \text{you}, \text{enjoy}, \text{often}\}$ is associated with the following embedding matrix:

$E = \begin{bmatrix} 0 & 1 \\ 1 & 1 \\ 1 & 0 \\ -1 & 1 \end{bmatrix}$

Words are indexed in the order shown above: “tea” (index 0), “you” (1), “enjoy” (2), and “often” (3). No positional encodings are used.

The parameters of the attention layer are:

$W_Q = \begin{bmatrix} 0.5 & 1 \\ 1 & 0.5 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 1 & 0.2 \\ 0.5 & 1 \end{bmatrix}, \quad W_O = \begin{bmatrix} 0.5 & 0.5 \\ 1 & -1 \end{bmatrix}$

For the input sequence “you enjoy tea often”. The computed attention matrix $A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)$ is given as:

$A = \begin{bmatrix} 0.55 & 0.32 & 0.11 & 0.02 \\ 0.43 & 0.25 & 0.21 & 0.1 \\ 0.39 & 0.39 & 0.16 & 0.07 \\ 0.24 & 0.4 & 0.2 & 0.17 \end{bmatrix}$

Answer the given subquestions. If you believe the information is insufficient, enter -1 as your answer.

For the input “you enjoy tea often”, compute the final representation of the word “tea” after the attention layer (i.e., after applying WQ, WK, WV , attention weights, and WO). Enter the sum of the elements in the resulting vector.

A vocabulary $\mathcal{V} = \{\text{tea}, \text{you}, \text{enjoy}, \text{often}\}$ is associated with the following embedding matrix:

$E = \begin{bmatrix} 0 & 1 \\ 1 & 1 \\ 1 & 0 \\ -1 & 1 \end{bmatrix}$

Words are indexed in the order shown above: “tea” (index 0), “you” (1), “enjoy” (2), and “often” (3). No positional encodings are used.

The parameters of the attention layer are:

$W_Q = \begin{bmatrix} 0.5 & 1 \\ 1 & 0.5 \end{bmatrix}, \quad W_K = \begin{bmatrix} 1 & 0.5 \\ -0.5 & 1 \end{bmatrix}, \quad W_V = \begin{bmatrix} 1 & 0.2 \\ 0.5 & 1 \end{bmatrix}, \quad W_O = \begin{bmatrix} 0.5 & 0.5 \\ 1 & -1 \end{bmatrix}$

For the input sequence “you enjoy tea often”. The computed attention matrix $A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)$ is given as:

$A = \begin{bmatrix} 0.55 & 0.32 & 0.11 & 0.02 \\ 0.43 & 0.25 & 0.21 & 0.1 \\ 0.39 & 0.39 & 0.16 & 0.07 \\ 0.24 & 0.4 & 0.2 & 0.17 \end{bmatrix}$

Answer the given subquestions. If you believe the information is insufficient, enter -1 as your answer.

Suppose the input sentence is “tea you enjoy often”. Using the same matrices and processing method, what is the attention score (i.e., softmax entry from A) for the query word “tea” attending to key word “often”?

What percentage of entries in the attention matrix are zero (i.e., the sparsity)?

Consider the short sentence:

“large language models are cool”

A transformer processes this sequence using the naive relative positional embedding method. The transformer's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 5$. We use zero-based indexing for token positions:

$0: \texttt{large} \quad 1: \texttt{language} \quad 2: \texttt{models} \quad 3: \texttt{are} \quad 4: \texttt{cool}$

Note: In this context, the relative position (k) between two tokens is:

$\text{Relative position} = j - i$

rows $=$ current token $i$, cols $=$ other token $j$

Positive values mean the other token is after the current token, negative values mean it is before, and 0 means it is the same token.

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \end{bmatrix}$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position $k \in \mathbb{Z}$,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10}\right].$

Based on the above data, answer the given subquestions.

Consider the short sentence:

“large language models are cool”

A transformer processes this sequence using the naive relative positional embedding method. The transformer's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 5$. We use zero-based indexing for token positions:

$0: \texttt{large} \quad 1: \texttt{language} \quad 2: \texttt{models} \quad 3: \texttt{are} \quad 4: \texttt{cool}$

Note: In this context, the relative position (k) between two tokens is:

$\text{Relative position} = j - i$

rows $=$ current token $i$, cols $=$ other token $j$

Positive values mean the other token is after the current token, negative values mean it is before, and 0 means it is the same token.

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \end{bmatrix}$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position $k \in \mathbb{Z}$,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10}\right].$

Based on the above data, answer the given subquestions.

Consider the short sentence:

“large language models are cool”

A transformer processes this sequence using the naive relative positional embedding method. The transformer's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 5$. We use zero-based indexing for token positions:

$0: \texttt{large} \quad 1: \texttt{language} \quad 2: \texttt{models} \quad 3: \texttt{are} \quad 4: \texttt{cool}$

Note: In this context, the relative position (k) between two tokens is:

$\text{Relative position} = j - i$

rows $=$ current token $i$, cols $=$ other token $j$

Positive values mean the other token is after the current token, negative values mean it is before, and 0 means it is the same token.

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \end{bmatrix}$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position $k \in \mathbb{Z}$,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10}\right].$

Based on the above data, answer the given subquestions.

With a naive implementation, for each token embedding, we need a way to combine (say, addition) relative position embeddings. For token index $i \in \{0, \ldots, T-1\}$,

$\boldsymbol{h}_i = \boldsymbol{x}_i + \sum_{j=0}^{T-1} \boldsymbol{p}_{j-i}.$

Using the naive formula above, compute the final combined embedding $\boldsymbol{h}_i$ for the token "language" (i.e., $i = 1$). Submit the sum of all the elements.

Consider the short sentence:

“large language models are cool”

A transformer processes this sequence using the naive relative positional embedding method. The transformer's embedding dimension is $d_{\text{model}} = 4$, and the number of tokens is $T = 5$. We use zero-based indexing for token positions:

$0: \texttt{large} \quad 1: \texttt{language} \quad 2: \texttt{models} \quad 3: \texttt{are} \quad 4: \texttt{cool}$

Note: In this context, the relative position (k) between two tokens is:

$\text{Relative position} = j - i$

rows $=$ current token $i$, cols $=$ other token $j$

Positive values mean the other token is after the current token, negative values mean it is before, and 0 means it is the same token.

1. Token embeddings (rows correspond to tokens in order):

   $\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 \\ 4 & 4 & 4 & 4 \\ 5 & 5 & 5 & 5 \end{bmatrix}$

   Denote the $i$-th row by $\boldsymbol{x}_i$.

2. Positional embedding definition: For a relative position $k \in \mathbb{Z}$,

   $\boldsymbol{p}_{(j-i)} = \left[\frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10} \quad \frac{(j-i)}{10}\right].$

Based on the above data, answer the given subquestions.