Deep Learning Quiz 2: 1 December 2024 (September 2024 term)

Given a quadratic loss function $L(w) = w^2$, where $w$ represents the model parameter, you are using the AdaGrad optimizer with Stochastic Gradient Descent to minimize this loss. The learning rate $\eta$ is set to 1, and at the initial iteration ($t=0$), the parameter $w$ has a starting value of $w=2$.

AdaGrad Update Rule:

$v_t = v_{t-1} + (\nabla w_t)^2$

$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t$

$v_{-1} = 0$

$\text{use } \epsilon = 0$

Based on the above data, answer the given subquestions.

Given a quadratic loss function $L(w) = w^2$, where $w$ represents the model parameter, you are using the AdaGrad optimizer with Stochastic Gradient Descent to minimize this loss. The learning rate $\eta$ is set to 1, and at the initial iteration ($t=0$), the parameter $w$ has a starting value of $w=2$.

AdaGrad Update Rule:

$v_t = v_{t-1} + (\nabla w_t)^2$

$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t$

$v_{-1} = 0$

$\text{use } \epsilon = 0$

Based on the above data, answer the given subquestions.

Given a quadratic loss function $L(w) = w^2$, where $w$ represents the model parameter, you are using the AdaGrad optimizer with Stochastic Gradient Descent to minimize this loss. The learning rate $\eta$ is set to 1, and at the initial iteration ($t=0$), the parameter $w$ has a starting value of $w=2$.

AdaGrad Update Rule:

$v_t = v_{t-1} + (\nabla w_t)^2$

$w_{t+1} = w_t - \frac{\eta}{\sqrt{v_t + \epsilon}} * \nabla w_t$

$v_{-1} = 0$

$\text{use } \epsilon = 0$

Based on the above data, answer the given subquestions.

Which of the following optimization algorithms adapts the learning rate based on the current gradient and the history of gradients?

Which of the following are regularization technique(S)?

Which of the following activation function(s) is (are) prone to vanishing gradient problem?

Which optimization algorithm combines the advantages of RM-Sprop and momentum?

What is the key reason for the development of adaptive optimization algorithms like RMSprop and Adam?

What is the derivative of the Leaky ReLU activation function with respect to its input?

Which of the following options represents the correct ordering in terms of the bias of the models illustrated in the images below?

Which of the following is a common form of regularization that involves adding a penalty proportional to the square of the weights?

Consider an input image of shape 10×10 with 3 channels of R-G-B, is processed using one of the following neural architectures:
• Fully-connected layer (FC) with 2 neurons (Note: The input volume is “flattened” into a column vector before being fed into the layer.)
• Convolutional layer (CONV) with three filters of size 2 × 2, a stride of 2 is used, and no padding is applied.
Based on the above data answer the given subquestions.

Which architecture has fewer parameters (CONV or FC)?

Consider an input image of shape 10×10 with 3 channels of R-G-B, is processed using one of the following neural architectures:
• Fully-connected layer (FC) with 2 neurons (Note: The input volume is “flattened” into a column vector before being fed into the layer.)
• Convolutional layer (CONV) with three filters of size 2 × 2, a stride of 2 is used, and no padding is applied.
Based on the above data answer the given subquestions.

What is the difference in the number of trainable parameters between these two neural architectures?

Consider an input image of shape 10×10 with 3 channels of R-G-B, is processed using one of the following neural architectures:
• Fully-connected layer (FC) with 2 neurons (Note: The input volume is “flattened” into a column vector before being fed into the layer.)
• Convolutional layer (CONV) with three filters of size 2 × 2, a stride of 2 is used, and no padding is applied.
Based on the above data answer the given subquestions.

Which of the following statements is/are true about a CONV layer?

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.

Consider the below deep neural architecture, being applied on an image of size 128 × 128 with 3 channels of R-G-B.

Assume
i For convolution layers, a stride of 1 is used, and no padding is applied.
ii There is no bias associated in any layer just as we did in our course
iii Each time we slide the filter we get one value in the output. The resulting output is called a feature map
iv Suppose Width, Height and Depth (number of channels) are denoted by W,H,D respectively, then the “Feature map volume” of the tensor (3-D array) = W × H × D (multiplication of W,H,D). Based on the above data answer the given subquestions.