Question 11
Consider a fully connected neural network as follows:
•
One input neuron
•
A single hidden layer with two neurons
•
One output neuron.
The network uses the ReLU activation function in both the hidden and output layers. The loss function used is the squared error:
where
is the true target and
is the network output.
Let the initial weights of the network be
.
Based on the above data, answer the given subquestions.
Suppose the network uses the ELU activation function with
in the hidden layer and ReLU activation function in the output layer. Using the same initial weights, compute the updated value
of
after one step of stochastic gradient descent on the data point
with a
learning rate of
. Enter the answer correct to one decimal place.
Hint: ELU activation function is given by