Deep Learning Quiz 1: 16 July 2023 (May 2023 term)

Consider the data points $\left(x_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, x_2 = \begin{bmatrix} 1 \\ -1 \end{bmatrix}, x_3 = \begin{bmatrix} -1 \\ 1 \end{bmatrix}, x_4 = \begin{bmatrix} -1 \\ -1 \end{bmatrix}\right)$.

Of these, the points ($x_1, x_2$ and $x_3$) belong to positive (1) class and the point $x_4$ belongs to negative (0) class. The perceptron uses the following decision rule

$\hat{y} = \begin{cases} 1, & \text{if } w^T x \geq 0 \\ 0, & w^T x < 0 \end{cases}$

Based on the above data, answer the given subquestions.

Suppose that the decision boundary passes through the origin and through the point $D = \begin{bmatrix} 0.5 \\ 1 \end{bmatrix}$. The weight vector is initialized to $w = \begin{bmatrix} w_0 \\ w_1 \end{bmatrix}$ with $w_0 = -1$. What is the value of $w_1$?

Consider the data points $\left(x_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, x_2 = \begin{bmatrix} 1 \\ -1 \end{bmatrix}, x_3 = \begin{bmatrix} -1 \\ 1 \end{bmatrix}, x_4 = \begin{bmatrix} -1 \\ -1 \end{bmatrix}\right)$.

Of these, the points ($x_1, x_2$ and $x_3$) belong to positive (1) class and the point $x_4$ belongs to negative (0) class. The perceptron uses the following decision rule

$\hat{y} = \begin{cases} 1, & \text{if } w^T x \geq 0 \\ 0, & w^T x < 0 \end{cases}$

Based on the above data, answer the given subquestions.

How many points were correctly classified?

Consider the data points $\left(x_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, x_2 = \begin{bmatrix} 1 \\ -1 \end{bmatrix}, x_3 = \begin{bmatrix} -1 \\ 1 \end{bmatrix}, x_4 = \begin{bmatrix} -1 \\ -1 \end{bmatrix}\right)$.

Of these, the points ($x_1, x_2$ and $x_3$) belong to positive (1) class and the point $x_4$ belongs to negative (0) class. The perceptron uses the following decision rule

$\hat{y} = \begin{cases} 1, & \text{if } w^T x \geq 0 \\ 0, & w^T x < 0 \end{cases}$

Based on the above data, answer the given subquestions.

Update the weight values using the perceptron learning algorithm by visiting the following points in order (x4, x3). What is the angle (in degrees) between the updated weight vector and the initial weight vector?

Consider the data points $\left(x_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, x_2 = \begin{bmatrix} 1 \\ -1 \end{bmatrix}, x_3 = \begin{bmatrix} -1 \\ 1 \end{bmatrix}, x_4 = \begin{bmatrix} -1 \\ -1 \end{bmatrix}\right)$.

Of these, the points ($x_1, x_2$ and $x_3$) belong to positive (1) class and the point $x_4$ belongs to negative (0) class. The perceptron uses the following decision rule

$\hat{y} = \begin{cases} 1, & \text{if } w^T x \geq 0 \\ 0, & w^T x < 0 \end{cases}$

Based on the above data, answer the given subquestions.

Consider a sigmoid neuron that takes in an input vector $x = \begin{bmatrix} 1 \\ -1 \\ 1 \\ -1 \end{bmatrix}$. The weight vector $\theta$ is initialized to $\begin{bmatrix} -1 \\ 1 \\ -1 \\ 1 \end{bmatrix}$. The output from the sigmoid neuron is

$\hat{y} = \frac{1}{1 + exp(-\theta^T x)}$

Suppose we use the following loss function

$L = \frac{1}{2}(y - \hat{y})^2$

Update the weight vector once by running the Gradient Descent algorithm with $\eta = 20$. Assume the true label $y = 1$. What is the new loss value (that is, the loss computed after updating the weight vector)?

Look at the contours of a hill shown below. Suppose a person walks from location A to location C via location B. Then choose the correct statements from the following list of statements

Consider a neural network with three hidden layers and one output layer. The hidden layers contain 100 neurons each. Suppose we have a square image of size 30×30 containing either a cat (positive class) or a dog (negative class). The neural network is designed to recognize it by outputting a probability score for each class. The input image is flattened into an array of size 900. Assume that all neurons in the network have bias associated with them and use the sigmoid activation function. How many parameters are there in the network?

Based on the above data, answer the given subquestions.

Suppose that the neural network is used for a classification problem with 1000 classes (such that any given input belongs to exactly one class). Suppose further that the number of neurons in the second hidden layer is 3 and the number of neurons in the first hidden layer is also 3. The weights of $W_3$ are all initialized to 1. The input $h_0 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}$. The other weights $W_1$ and $W_2$ are initialized as follows,

$W_1 = W_2 = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 0 \end{bmatrix}$

The true label $y = 500$ is one hot encoded as $y = \begin{bmatrix} 0 & 0 & \cdots & 0 & 1 & 0 & 0 & 0 & \cdots & 0 \end{bmatrix}^T$. Assume that the output layer uses softmax activation and the neurons in the network has no bias associated with it. Compute the cross entropy loss.

Note:If you think the given information is not sufficient to calculate the loss, then enter -1 as answer.

Based on the above data, answer the given subquestions.

Assume we have only three neurons in the output layer with a softmax activation function. Suppose that the weights are initialized as follows

$W_1 = W_2 = W_3 = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 0 \end{bmatrix}. \text{ Moreover,}$

$a_1 = \begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix} \quad a_2 = \begin{bmatrix} 3.22 \\ 2.92 \\ 3.22 \end{bmatrix} \quad a_3 = \begin{bmatrix} 3.84 \\ 3.82 \\ 3.82 \end{bmatrix} \quad y = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$

Use cross entropy loss and compute the gradient of $w_{10}$ (that is, $\nabla w_{10}$) of $W_3$. (More precisely, $\nabla w_{310}$)

Based on the above data, answer the given subquestions.

Assume we have only three neurons in the output layer with a softmax activation function. Suppose that the weights are initialized as follows

$W_1 = W_2 = W_3 = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 0 \end{bmatrix}. \text{ Moreover,}$

$a_1 = \begin{bmatrix} 2 \\ 1 \\ 1 \end{bmatrix} \quad a_2 = \begin{bmatrix} 3.22 \\ 2.92 \\ 3.22 \end{bmatrix} \quad a_3 = \begin{bmatrix} 3.84 \\ 3.82 \\ 3.82 \end{bmatrix} \quad y = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$

Use cross entropy loss and compute the gradient of $w_{10}$ (that is, $\nabla w_{10}$) of $W_2$. (More precisely, $\nabla w_{210}$)

Suppose that a team solves a 1000-class classification problem using a neural network that contains 32 layers. Assume that the training set contains exactly 32 samples per class. The team creates batches of samples, each of size 32. The batch is created such that all the samples in a batch belongs to the same class.
Based on the above data, answer the given subquestions.

Suppose that the team trains the neural network with the given configuration using mini-batch Gradient Descent algorithm for 32 epochs. Then how many times do the parameters of the network get updated?

Suppose that a team solves a 1000-class classification problem using a neural network that contains 32 layers. Assume that the training set contains exactly 32 samples per class. The team creates batches of samples, each of size 32. The batch is created such that all the samples in a batch belongs to the same class.
Based on the above data, answer the given subquestions.

Suppose we use SGD (Stochastic Gradient Descent) algorithm to update the parameters of the network by running it for 32 epochs. Then how many times do the parameters of the network get updated?