Here is a configuration of the GPTNeo model from the Hugging Face hub. "bos_token_id": 50256, "classifier_dropout": 0.1, "embed_dropout": 0.0, "eos_token_id": 50256, "hidden_size": 2048, "initializer_range": 0.02, "intermediate_size": null, "layer_norm_epsilon": 1e-05, "max_position_embeddings": 2048, "model_type": "gpt_neo", "num_heads": 16, "num_layers": 24, "resid_dropout": 0.0, "transformers_version": "4.44.2", "use_cache": true, "vocab_size": 50257, "window_size": 256 Figure 1: GPTNeoConfig Based on the above data, answer the given subquestions. Enter the number of parameters in the embedding layer of the model in millions. For example, if the answer is 1234567. Then enter it as 1.23

Deep Learning Practice · Quiz 1 · 27 Oct 2024 · September 2024 term

Question 11: Here is a configuration of the GPTNeo model from the Hug…

Question 11

+3 marksNumerical answer

Here is a configuration of the GPTNeo model from the Hugging Face hub.

text

1"bos_token_id": 50256,
2"classifier_dropout": 0.1,
3"embed_dropout": 0.0,
4"eos_token_id": 50256,
5"hidden_size": 2048,
6"initializer_range": 0.02,
7"intermediate_size": null,
8"layer_norm_epsilon": 1e-05,
9"max_position_embeddings": 2048,
10"model_type": "gpt_neo",
11"num_heads": 16,
12"num_layers": 24,
13"resid_dropout": 0.0,
14"transformers_version": "4.44.2",
15"use_cache": true,
16"vocab_size": 50257,
17"window_size": 256

Figure 1: GPTNeoConfig

Based on the above data, answer the given subquestions.

Enter the number of parameters in the embedding layer of the model in millions. For example, if the answer is 1234567. Then enter it as 1.23

Show answer

Correct answer: 102.5 (accepted within ±0.5)

See the whole paper Take it as a mock test

Question 11 of 15 in the IIT Madras BS Deep Learning Practice (Deep Learning Practice) Quiz 1 paper sat on 27 Oct 2024, in the September 2024 term (IIT M DEGREE AN EXAM QDB2 27 Oct 2024). It carries 3 marks.

Question 11: Here is a configuration of the GPTNeo model from the Hug…

Question 11

More questions from this paper