Large Language Models Quiz 1: 23 February 2025 (January 2025 term)

Consider following assertion and reason pair:
Assertion: A transformer model cannot be trained in autogregressive mode for machine translation task.
Reason: Teacher forcing improves the convergence rate.
Choose the correct statements

Suppose we use a pre-trained model for text generation with the given prompt “I am going to”. Which of the following decoding strategies can be used such that the pre-trained model generates different text completion each time it is executed

Which of the following models, in general, struggle to encode a context when translating long sentences?

Choose the correct statements

Which of the following decoding strategies is (are) appropriate for machine translation tasks? Assume we have infinite compute and the values for k ≥ 3, p > 0.35 where required.

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

• Source vocabulary size $= 100$
• Target vocabulary size $= 200$
• maximum sequence length $= 32$
• length of context window $(T)$ for both encoder and decoder $= 32$
• number of heads $n_h = 4$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

Suppose the number of learnable parameters in the source input-embedding layer is 3200, how many parameters are there in the positional embedding layer of the source language?

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

• Source vocabulary size $= 100$
• Target vocabulary size $= 200$
• maximum sequence length $= 32$
• length of context window $(T)$ for both encoder and decoder $= 32$
• number of heads $n_h = 4$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

How many parameters are there in the multi-head attention layer of the encoder (exclude the parameters in the WO matrix used for linear transformation and FFN layer)?

Consider the following configuration for the Vannila transformer architecture with one encoder layer and one decoder layer.

• Source vocabulary size $= 100$
• Target vocabulary size $= 200$
• maximum sequence length $= 32$
• length of context window $(T)$ for both encoder and decoder $= 32$
• number of heads $n_h = 4$
• $dff = 4 * dmodel$
• $dq = dk = dv = \frac{dmodel}{n_h}$

Based on the above data, answer the given subquestions.

How many parameters does the matrix WQ have in anyone head of the encoder?

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 & 8 \\ 4 & 7 & 11 & 14 \end{bmatrix}$

It has 4 samples with 2 features each i.e. batch size is 4. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1, \beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Say matrix F is computed by applying batch normalization on X, what will be sum of every element in first / topmost row of F?

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 & 8 \\ 4 & 7 & 11 & 14 \end{bmatrix}$

It has 4 samples with 2 features each i.e. batch size is 4. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1, \beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Say matrix F is computed by applying layer normalization on X, what will be sum of every element in first / leftmost column of F?

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 & 8 \\ 4 & 7 & 11 & 14 \end{bmatrix}$

It has 4 samples with 2 features each i.e. batch size is 4. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1, \beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Consider following input matrix to a single neural network layer:

$X = \begin{bmatrix} 1 & 2 & 3 & 8 \\ 4 & 7 & 11 & 14 \end{bmatrix}$

It has 4 samples with 2 features each i.e. batch size is 4. Each column represents a sample. Left most column represents the first sample and topmost row represents the first feature vector.

Note: assume $\gamma = 1, \beta = 0$ and $\epsilon = 10^{-6}$ for both batch and layer normalization, if not mentioned explicitly. Use standard scaling.

Based on the above data, answer the given subquestions.

Consider following corpus (in italics), which is used train a GPT like (i.e. decoder only) model for causal language modeling:

Sachin Tendulkar is called God of Cricket. Sachin has broken many records. Sachin has highest number of centuries.

Sachin Pilgaonkar is a famous actor in Marathi cinema. Sachin has acted in, directed and produced numerous films. Sachin has broken many stereotypes.

Assume each word in the above corpus is a token. Tokens are case insensitive i.e. “The” is same as “the”. Ignore special characters (e.g. period (.), semi colon (;), etc.)

Note: Estimate the joint probabilities using the Bayesian approach (no independence assumption) for the given corpus.

$x_i$ represents $i^{th}$ token from left to right. Tokens appear in italics.

Answer the given sub questions:

Consider following corpus (in italics), which is used train a GPT like (i.e. decoder only) model for causal language modeling:

Sachin Tendulkar is called God of Cricket. Sachin has broken many records. Sachin has highest number of centuries.

Sachin Pilgaonkar is a famous actor in Marathi cinema. Sachin has acted in, directed and produced numerous films. Sachin has broken many stereotypes.

Assume each word in the above corpus is a token. Tokens are case insensitive i.e. “The” is same as “the”. Ignore special characters (e.g. period (.), semi colon (;), etc.)

Note: Estimate the joint probabilities using the Bayesian approach (no independence assumption) for the given corpus.

$x_i$ represents $i^{th}$ token from left to right. Tokens appear in italics.

Answer the given sub questions:

Consider following corpus (in italics), which is used train a GPT like (i.e. decoder only) model for causal language modeling:

Sachin Tendulkar is called God of Cricket. Sachin has broken many records. Sachin has highest number of centuries.

Sachin Pilgaonkar is a famous actor in Marathi cinema. Sachin has acted in, directed and produced numerous films. Sachin has broken many stereotypes.

Assume each word in the above corpus is a token. Tokens are case insensitive i.e. “The” is same as “the”. Ignore special characters (e.g. period (.), semi colon (;), etc.)

Note: Estimate the joint probabilities using the Bayesian approach (no independence assumption) for the given corpus.

$x_i$ represents $i^{th}$ token from left to right. Tokens appear in italics.

Answer the given sub questions:

Consider following corpus (in italics), which is used train a GPT like (i.e. decoder only) model for causal language modeling:

Sachin Tendulkar is called God of Cricket. Sachin has broken many records. Sachin has highest number of centuries.

Sachin Pilgaonkar is a famous actor in Marathi cinema. Sachin has acted in, directed and produced numerous films. Sachin has broken many stereotypes.

Assume each word in the above corpus is a token. Tokens are case insensitive i.e. “The” is same as “the”. Ignore special characters (e.g. period (.), semi colon (;), etc.)

Note: Estimate the joint probabilities using the Bayesian approach (no independence assumption) for the given corpus.

$x_i$ represents $i^{th}$ token from left to right. Tokens appear in italics.

Answer the given sub questions:

Once the model is sufficiently trained, which of the following tokens will have the highest probability to be the next token, if the input is “Sachin has”: