Question 11
The evolution of NLP models shows a distinct shift from task-specific architectures to a ”pre-train, finetune” paradigm, and now towards large-scale, general-purpose models. Which of the following accurately represents this evolution and the capabilities at each stage? (Select ALL that apply)
The earliest models (e.g., n-grams) were statistical, required task-specific design, and had limited generalization capacity.
The ”pre-train, fine-tune” era (e.g., BERT, GPT) introduced transfer learning, where a model was first trained on a general language task and then fully adapted to a specific downstream task.
Modern Large Language Models (LLMs like GPT-4) exhibit ”emerging abilities,” allowing them to perform new tasks with zero or few examples (in-context learning) without any weight updates.
Word2vec was a complete language model capable of generating text, similar to GPT.
The primary innovation of transformers over RNNs was the use of recurrent connections, which made them more efficient to train on parallel hardware.