Deep Learning Quiz 2: 6 August 2023 (May 2023 term)

Suppose that a neural network has millions of parameters (weights and biases). A team decides to use an optimization algorithm with a learning rate scheme that is local to each individual parameters in the network. Moreover, the learning rate changes in each iteration based on the magnitude of gradients pertaining to a parameter in the past. Which of the following optimization algorithms satisfy the team’s requirements?

Suppose a data set has $N$ samples and each sample has two features $f_1 \in \{0, 1\}$ and $f_2 \in \{0, 1\}$. Assume that $f_1$ is a sparse feature and $f_2$ is a dense feature (that is, $f_1$ value for most of the samples is zero). Further, we apply Stochastic Gradient Descent on this data and plot a contour plot of the loss surface and observe the movement of parameters (i.e., trajectory) across iterations. Let $w_1$ and $w_2$ be the parameters corresponding to $f_1$ and $f_2$ respectively.

Assume bias to be zero for this question and the parameters are initialized to zero.

In the contour plot of a loss surface, what is the initial movement expected to look like if $w_1$ is plotted on the horizontal axis and $w_2$ is plotted on the vertical axis?

Hint: Think of the possible configurations of inputs for the first few iterations

We know that the weight update rule for gradient descent algorithm is given by

$w_{t+1} = w_t - \eta_t \nabla w_t$

which is equivalently written as

$w_{t+1} = w_0 - \eta_t \sum_{i=1}^{t} \nabla w_i$

Choose all the correct bounds

Which of the following activation function(s) is (are) prone to vanishing gradient problem?

An input image of dimension 35 × 35 × 3 is to be convolved with 20 filters (kernels) of spatial dimension 3 × 3. In all the sub-questions, assume a bias term for kernels.

Suppose that we pad zeros of size 2 (p = 2) around the sides of the image (in all channels). Assume stride 2 for the convolution operation. All the 20 filters have bias associated with it. What is the dimension of the output volume? Enter the number of elements in the output volume.

An input image of dimension 35 × 35 × 3 is to be convolved with 20 filters (kernels) of spatial dimension 3 × 3. In all the sub-questions, assume a bias term for kernels.

How many numbers of parameters are there in the convolution layer?

An input image of dimension 35 × 35 × 3 is to be convolved with 20 filters (kernels) of spatial dimension 3 × 3. In all the sub-questions, assume a bias term for kernels.

Suppose we flatten the output of convolution layer, that is $\mathbb{R}^{m \times n \times k} \to \mathbb{R}^{m \cdot n \cdot k \times 1}$ and apply 1-D kernel of dimension $10 \times 1$ (no zero padding, stride=5).

The result of this convolution operation is then connected to a fully connected layer of dimension $10 \times 1$ with a weight matrix $W$ and a bias $b$. How many parameters are there in the ENTIRE network then?

The input matrix $X$ and the kernels $(K, F)$ are given below.

The bias value for both kernels is initialized at 0.5

$X = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}, \quad F = \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}$

In the subsequent questions, assume no padding and stride one wherever necessary.

Please be careful in your calculations as the questions are linked

Create two feature maps by convolving the kernels K, F over the input matrix X and stack the outputs to create a tensor as shown below. The dimension of A is m x n x 2. The first feature map of size m x n is the output for K and the second feature map of size m x n is for F. What is the sum of the elements in the tensor A?

The input matrix $X$ and the kernels $(K, F)$ are given below.

The bias value for both kernels is initialized at 0.5

$X = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}, \quad F = \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}$

In the subsequent questions, assume no padding and stride one wherever necessary.

Please be careful in your calculations as the questions are linked

The input matrix $X$ and the kernels $(K, F)$ are given below.

The bias value for both kernels is initialized at 0.5

$X = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}, \quad F = \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}$

In the subsequent questions, assume no padding and stride one wherever necessary.

Please be careful in your calculations as the questions are linked

The input matrix $X$ and the kernels $(K, F)$ are given below.

The bias value for both kernels is initialized at 0.5

$X = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}, \quad F = \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}$

In the subsequent questions, assume no padding and stride one wherever necessary.

Please be careful in your calculations as the questions are linked

The input matrix $X$ and the kernels $(K, F)$ are given below.

The bias value for both kernels is initialized at 0.5

$X = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 0 & 1 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}, \quad F = \begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}$

In the subsequent questions, assume no padding and stride one wherever necessary.

Please be careful in your calculations as the questions are linked

Enter the sum of updated bias (exclude weights).