Question 2: Suppose a data set has N samples and each sample has two …

Suppose a data set has $N$ samples and each sample has two features $f_1 \in \{0, 1\}$ and $f_2 \in \{0, 1\}$. Assume that $f_1$ is a sparse feature and $f_2$ is a dense feature (that is, $f_1$ value for most of the samples is zero). Further, we apply Stochastic Gradient Descent on this data and plot a contour plot of the loss surface and observe the movement of parameters (i.e., trajectory) across iterations. Let $w_1$ and $w_2$ be the parameters corresponding to $f_1$ and $f_2$ respectively.

Assume bias to be zero for this question and the parameters are initialized to zero.

In the contour plot of a loss surface, what is the initial movement expected to look like if $w_1$ is plotted on the horizontal axis and $w_2$ is plotted on the vertical axis?

Hint: Think of the possible configurations of inputs for the first few iterations