Question 1

Consider the following two statements regarding model performance:\ **Statement 1:** A model achieving zero training loss is guaranteed to perform well on unseen data. **Statement 2:** Incorporating a regularization term in the loss function may lead to higher training loss but lower generalization error.\ Which of the following options is correct? How does unsupervised layerwise pretraining help in alleviating the vanishing gradient problem? A dataset is given by $$X = \begin{bmatrix} 1 & 2 & 0 & 5 & 0 \\ 1 & 0 & 3 & 0 & 0 \\ 1 & 1 & 0 & 1 & 1 \\ 1 & -1 & 0 & 0 & 0 \end{bmatrix}, y = \begin{bmatrix} 10 \\ 6 \\ 5 \\ 2 \end{bmatrix}$$ The rows of $X$ represent samples and the columns represent features, with the first column corresponds the bias term. We use a linear regression neuron where the prediction $\hat{y}_i$ for a sample $x_i$ is given by the linear combination $\hat{y}_i = z_i = \sum_{j=0}^{4} w_j x_{ij}$. The weights are updated using Stochastic Gradient Descent (SGD) for one epoch (i.e., once for each of the 4 samples). The loss function is the Mean Squared Error, $L = (\hat{y} - y)^2$. If all weights are initialized to $w_j = 0.5$, which of the following weights is updated the *fewest* number of times?