Introduction to Natural Language Processing End Term: 22 December 2024 (September 2024 term)

What are the examples of a free morpheme?

Which of the following are challenges in applying Shapley values for model explainability?

Consider top-k sampling with $k = 4$ as the decoding strategy for a large language model. For the context "I want to become a", the probabilities output by the model over all the words in the vocabulary are extracted and sorted in descending order. The top four words and their associated probabilities are given below:

• doctor, 0.4
• lawyer, 0.3
• writer, 0.06
• cook, 0.04

If a word is sampled at this stage, what is the probability with which the model outputs "lawyer"?

In a transformer model, attention rollout is used to compute the contribution of input tokens to the final output. Consider a model with 3 layers and an input sequence of 4 tokens $t_1, t_2, t_3, t_4$. The attention scores for each token in each layer are:

• Layer 1: $[0.2, 0.3, 0.4, 0.1]$
• Layer 2: $[0.3, 0.2, 0.3, 0.2]$
• Layer 3: $[0.4, 0.3, 0.2, 0.1]$

The initial contributions for each token $t_1, t_2, t_3, t_4$ are equal:

$[0.25, 0.25, 0.25, 0.25]$

Using attention rollout, compute the final contribution of token $t_1$ to the model's output. Enter your answer correct to three decimal points.

FastText is a word embedding model that extends Word2Vec by considering subword information through character n-grams (e.g., bi-grams, tri-grams). Which of the following statements about FastText's behavior and capabilities are true?

The Direct Preference Optimization (DPO) loss function is given as:

$L_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim D}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$

where,

• $\pi_\theta(y \mid x)$: Probability assigned to an output $y$ by the trained policy (fine-tuned model) for a given input $x$.
• $\pi_{\text{ref}}(y \mid x)$: Probability assigned by the reference policy (pretrained model).
• $(x, y_w, y_l)$: Input $x$ with a preferred output $y_w$ and a less preferred output $y_l$.
• $\beta$: A temperature parameter controlling the sensitivity to log-probability ratios.
• $\sigma(z) = \frac{1}{1+e^{-z}}$: The logistic sigmoid function.

Which of the following statements about the terms in the DPO loss function are correct?

Which of the following are suitable choices for an autoregressive language model with the primary objective of using it to generate text based on a given prompt?
(1) Vanilla RNN
(2) Bidirectional RNN

What will be the 0th element in the positional encoding for a word at position 10000 in models with dimensions 512 and 1024, respectively?

Which of the following approaches would be the best fit for these specific use cases? 1. A system that retrieves the latest financial data to provide investment advice.
2. Optimizing a general-purpose model for sentiment analysis on product reviews.
3. Generating concise summaries from a pre-trained model with a carefully worded input. 4. Using a general-purpose language model without any modifications.

What are the inputs and outputs of the query encoder and passage encoder in the Dense Passage Retriever (DPR)?

Based on the above data, answer the given subquestions.

Which of the following is the embedding matrix?

Based on the above data, answer the given subquestions.

Which of the following is the matrix corresponding to the feedforward connections between the embedding layer and the hidden layer?

Based on the above data, answer the given subquestions.

Which of the following are true?

Based on the above data, answer the given subquestions.

Consider an encoder-decoder transformer used for story generation. If we use exhaustive search to generate tokens for 20 time steps for a story generation task given a vocabulary of size 5,000 , what is the size of the search space?

A decoder-only transformer-based language model is used to generate text based on some prompt. The language model is prompted ten times with the same prompt. It produces the same response all ten times. Each run is to be treated independently without any memory of the previous runs. Which of the following could be the most likely decoding strategy used?

In the context of bias mitigation techniques, loss function modification would fall under which of the following categories?

Which of the following techniques can be used to explain the predictions of a deep learning model by attributing the importance of each feature?

Why is calculating Shapley values computationally expensive for models with a large number of features?