Quiz Space

Deep Learning Practice · Quiz 1 · 23 Feb 2025 · January 2025 term

Question 2: Consider the following code where a tokenizer is created …

Question 2

+3 marksOne correct option

Consider the following code where a tokenizer is created using the BPE model:

python
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.normalizers import Lowercase
from tokenizers.pre_tokenizers import Whitespace


data = ["existent", "non", "exist", "word"]


# Create a tokenizer with BPE model without specifying the unk_token
model = BPE()
tokenizer = Tokenizer(model)


# Normalizer and Pre-tokenizer
tokenizer.normalizer = Lowercase()
tokenizer.pre_tokenizer = Whitespace()


# Train the tokenizer
trainer = BpeTrainer(vocab_size=5000, special_tokens=["<s>", "</s>", "<pad>"])
tokenizer.train_from_iterator(data, trainer)


# Access the tokenizer's vocabulary
vocab = tokenizer.get_vocab()


# Test tokenization with a word not present in the vocabulary


tokens = tokenizer.encode("nonexistentword").tokens
print("Tokens:", tokens)

Which of the following statements is correct about the tokenization output when the word "nonexistentword" is encountered?

  1. A

    The tokenizer will output the token [UNK] because "nonexistentword" is not in the vocabulary.

  2. B

    The tokenizer will output the word "nonexistentword" as a single token.

  3. C

    The tokenizer will output "non", "existent", "word" as separate tokens since it has split the word into known subwords.

  4. D

    The tokenizer will output "non", "exist", "ent", "word" as separate tokens since it has split the word into known subwords.

  5. E

    The tokenizer will output an error because "nonexistentword" is not present in the vocabulary and the unk_token was not defined.

Show answer

Correct answer

  • C

    The tokenizer will output "non", "existent", "word" as separate tokens since it has split the word into known subwords.

See the whole paperTake it as a mock test

Question 2 of 16 in the IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 23 Feb 2025, in the January 2025 term (IIT M DEGREE AN EXAM QDB2 23 Feb 2025). It carries 3 marks.

More questions from this paper

  1. Q1Figure question
  2. Q3Consider the following code snippet: What will likely be printed as the output?
  3. Q4Given the following code snippet: Which of the following options correctly adds LoRA to the base_model? (Whichever opti…
  4. Q5Given the following code for setting up the trainer: Which of the following lines correctly starts the fine-tuning proc…
  5. Q6What would happen if you set the target_modules parameter in the LoraConfig to an empty list []?
  6. Q7Consider the following code where the tokenizer is trained using two different vocabulary sizes: 5K and 50K. The functi…
  7. Q8What type of language modeling objective is used during the pretraining of GPT-2?
  8. Q9Which of the following is/are gradient based (fine tuning) methods?
  9. Q10Figure question
  10. Q11Select all the correct statements.
  11. Q12You are working with two datasets and trying to combine them using the following code: Assume the following: The ds1 da…
  12. Q13The original train split contains 25,000 samples, and 60% of the samples have a text length greater than 200. If the da…
  13. Q14Below is a snippet for loading the GPT-2 model configuration: the output of the above is Based on the above data, answe…
  14. Q15Below is a snippet for loading the GPT-2 model configuration: the output of the above is Based on the above data, answe…
  15. Q16Below is a snippet for loading the GPT-2 model configuration: the output of the above is Based on the above data, answe…