Deep Learning Quiz 1: 7 July 2024 (May 2024 term)

Consider a dataset of 100 points $\mathbf{x_1}, \mathbf{x_2}, \cdots, \mathbf{x_{100}}$.

First 50 points are $\mathbf{x_1} = \mathbf{x_2} = \cdots = \mathbf{x_{50}} = \begin{bmatrix} a \\ a \end{bmatrix}$ and next 50 points are $\mathbf{x_{51}} = \mathbf{x_{52}} = \cdots = \mathbf{x_{100}} = \begin{bmatrix} -a \\ -a \end{bmatrix}$, where $a > 0$. The first 50 data points belong to the positive class (denoted as 1) and the next 50 data points belong to the negative class (denoted by 0). Suppose that the perceptron learning algorithm is used to find the decision boundary that separates these data points with the following rule,

$f(\mathbf{x}) = \begin{cases} 1 & \text{if } \mathbf{w^T x} \ge 0 \\ 0 & \text{if } \mathbf{w^T x} < 0 \end{cases}$

The algorithm checks the data points in order. How often do the weights get updated until convergence? The weights do not include bias. If the algorithm does not converge, enter the answer as $-1$

Consider a single McCulloch-Pitts (MP) neuron with four binary inputs $x_1$, $x_2$, $x_3$, and $x_4$. The neuron produces an output $y$ based on a threshold function. The MP neuron uses the following decision rule

$\hat{y} = \begin{cases} 1, & \text{if } x_1 + x_2 + x_3 + x_4 > \theta \\ 0, & \text{otherwise} \end{cases}$

Given the following input combinations and their corresponding outputs:

Inputs: $x_1 = 1, x_2 = 0, x_3 = 1, x_4 = 1$ Output: $y = 1$
Inputs: $x_1 = 0, x_2 = 1, x_3 = 1, x_4 = 0$ Output: $y = 0$
Inputs: $x_1 = 1, x_2 = 1, x_3 = 0, x_4 = 1$ Output: $y = 1$

What minimum threshold value is required for the neuron to produce an output of 1? If the threshold can not be determined using the given information, enter the answer as $-1$.

Consider a feedforward neural network with one hidden layer trained using backpropagation for a binary classification task with classes labeled as 1 and 0. The network architecture is structured as follows:

• Input layer consisting of 5 neurons
• Hidden layer containing 3 neurons
• Output layer comprising 1 neuron

During the backpropagation process, the derivative of the sigmoid activation function $\sigma(z)$ with respect to its argument $z$ is given by:

$\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))$

If the loss function utilized for binary classification is the binary cross-entropy loss, and both the hidden layer and output layer use the sigmoid activation function, the cross-entropy loss is represented by:

$L(y, \hat{y}) = -y\log_2(\hat{y}) - (1 - y)\log_2(1 - \hat{y})$

Here, $\hat{y} = P(y = 1|\mathbf{x})$.

Given that the true label $y$ for a data point $\mathbf{x}$ is 1 and the predicted value $\hat{y}$ is 0.9, and the activation at the hidden layer is represented by $h_1 = \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}$, what is the value of $\frac{\partial L}{\partial W_{200}}$? Here, $W_{200}$ denotes the weight connecting the first neuron of the hidden layer to the output layer neuron.