Deep Learning Quiz 1: 25 February 2024 (January 2024 term)

Consider a neuron with binary inputs x1 and x2, and an output y. The neuron computes the weighted sum of its inputs and produces an output according to a threshold. The threshold is denoted as θ. The activation function is such that y = 1 if the weighted sum is greater than or equal to θ, otherwise y = 0.
Which of the following statements are correct regarding the neuron’s ability to represent logical AND and OR functions?

Suppose we have a perceptron with two inputs, x1 and x2. This perceptron undergoes training on a small dataset containing three points: (−1, 2) labeled as class 0, (0,−1) labeled as class 1, and (2, 1) labeled as class 0. The weights of the perceptron are initialized to zeros, and the model is trained until it reaches convergence. Given this scenario, what would be the assigned output class by the trained perceptron for the new point (−2, 0)?

You are training a neural network for sentiment analysis on a dataset of 10,000 text reviews. The dataset is divided into 80% for training and 20% for testing. You decide to use Minibatch Gradient Descent with a batch size of 32. If you perform a total of 100 epochs, how many parameter updates will be performed in total?

Let’s assume a continuous function f(x1,x2) is approximated using 3D tower function with a 3 hidden layer neural network using 100 towers. How many minimum number of neurons will we need to approximate the function f(x1,x2) ?

The logistic sigmoid neuron $\sigma(x)$ is defined as follows

$\sigma(x) = \frac{1}{1 + exp(-(wx + b))}$

where $w, b \in \mathbb{R}$ are learnable parameters.Take Mean Square Error loss where required

$L = 0.5 * (\hat{y} - y)^2$

Suppose we use the sigmoid function to fit the pair $x = 0, y = 1$, where $x$ is an input and $y$ is the ground truth. Suppose that $w$ is initialized to $w = 2$ and $b$ is initialized to $b = 1$.The prediction $\hat{y}$ by the model for the current $w, b$ is, $\hat{y} = 0.731$. Update the parameter once by keeping $\eta = 10$ and compute the loss. Enter the new loss value.

Note: Enter the loss value to three significant digits. That is, if your answer is 0.06134, then enter it as 0.061

Which of the following is true, given the optimal learning rate?

Based on the above data, answer the given subquestions.

How many parameters (including biases) are there in the entire network?

Based on the above data, answer the given subquestions.

Suppose that all elements in the input vector are zero and the corresponding true label is also 0. Further, suppose that all the parameters are initialized to zero.
What is the loss value if cross-entropy loss is used? Use natural logarithm ln.

Based on the above data, answer the given subquestions.

Assuming that all weights between layers h3 and O are initialized to one, with no bias associated with any neuron, what would be the computed cross-entropy loss for a given single data point? If the provided information is insufficient, please enter −1.

Consider a neural network with two hidden layers and one output layer, as shown below. Here, $a_k$ is a pre-activation and $h_k$ is the output from the $k-$th layer. All the neurons in the hidden layers use a sigmoid activation function. Suppose that the neural network is used for a classification problem with 3 classes (such that any given input belongs to exactly one class). Suppose further that the number of neurons in each hidden layer is 3 and we have only three neurons in the output layer. Suppose that the weights are initialized as follows

$W_1 = W_2 = W_3 = \begin{bmatrix} 1 & 1 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{bmatrix}$

The true label for the input $h_0 = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$ is one hot encoded as $y = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$.

Assume that the output layer uses softmax activation and the neurons in the network has no bias associated with it.

Based on the above data, answer the given subquestions.

Compute the cross entropy loss.
Note:If you think the given information is not sufficient to calculate the loss, then enter -1 as answer.

Consider a neural network with two hidden layers and one output layer, as shown below. Here, $a_k$ is a pre-activation and $h_k$ is the output from the $k-$th layer. All the neurons in the hidden layers use a sigmoid activation function. Suppose that the neural network is used for a classification problem with 3 classes (such that any given input belongs to exactly one class). Suppose further that the number of neurons in each hidden layer is 3 and we have only three neurons in the output layer. Suppose that the weights are initialized as follows

$W_1 = W_2 = W_3 = \begin{bmatrix} 1 & 1 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{bmatrix}$

The true label for the input $h_0 = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$ is one hot encoded as $y = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$.

Assume that the output layer uses softmax activation and the neurons in the network has no bias associated with it.

Based on the above data, answer the given subquestions.

Consider the following image:

As per your understanding of optimization algorithms, which of the following mappings will be correct (assume optimal learning rate)?