Reinforcement Learning Quiz 2: 4 August 2024 (May 2024 term)

The eligibility trace is used to store past observations and actions.

The eligibility trace determines the learning rate for updating the value estimates.

The eligibility trace helps in determining which state-action pairs are eligible for updates.

The eligibility trace maintains a history of rewards received in each state.

n-step TD prediction requires less memory and computational resources.

n-step TD prediction can handle delayed rewards and credit assignment over multiple time steps.

n-step TD prediction converges faster to the optimal policy.

n-step TD prediction guarantees convergence to the optimal value function for any choice of the learning rate.

Double Q-learning helps to reduce the variance in action value estimates, leading to more stable learning.

Double Q-learning is guaranteed to converge to the optimal policy even with non-linear function approximation.

Double Q-learning requires fewer samples to achieve the same level of performance compared to traditional Q-learning.

Double Q-learning is less sensitive to hyperparameter tuning, making it easier to train in complex environments.

The target network helps to reduce the computational complexity of the Q- learning updates.

The target network prevents the algorithm from overfitting to the current Q- value estimates.

The target network stabilizes the learning process by providing a more consistent target for Q-value estimation.

The target network enables the agent to explore more efficiently by incorporating additional sources of information.

Experience replay prioritizes important experiences based on their impact on the agent’s learning progress.

Experience replay reduces the computational complexity of the learning algorithm by storing only the most relevant experiences.

Experience replay mitigates the effects of non-stationarity in the data distribution by sampling uniformly from a replay buffer.

Experience replay breaks the temporal correlations in the data, allowing for more effective learning from past experiences

The choice of function approximator has no impact on the performance of policy gradient methods, as long as it can represent the policy space accurately.

Deep neural networks are the only suitable function approximators for policy gradient methods due to their ability to handle high-dimensional state and action spaces.

The choice of function approximator affects the stability, convergence rate, and generalization capabilities of policy gradient methods.

Linear function approximators are always preferred over non-linear function approximators for policy gradient methods to ensure faster convergence.

Incorporating a baseline reduces the variance of the gradient estimates, leading to more stable and efficient learning.

Incorporating a baseline increases the computational complexity of the algorithm, making it slower to converge.

Incorporating a baseline improves the exploration capabilities of the algorithm by encouraging more diverse action selection.

Incorporating a baseline ensures that the algorithm converges to the optimal policy more quickly by reducing the bias in the gradient estimates.

Decoupling the value function allows for more efficient computation of Q- values, reducing the computational complexity of the algorithm.

Decoupling the value function facilitates better exploration by explicitly separating the estimation of state values and advantage values.

Decoupling the value function enables Dueling DQN to learn state values more accurately, leading to improved stability during training.

Decoupling the value function allows Dueling DQN to better handle scenarios where certain actions are consistently advantageous or disadvantageous across states.

The semi-gradient method uses only a subset of the available data to compute parameter updates, while the full gradient method uses all available data.

The semi-gradient method updates the parameters of the value function based on a mixture of TD error and the gradient of the value function, while the full gradient method updates parameters using only the gradient.

The semi-gradient method is more computationally efficient than the full gradient method, requiring fewer iterations to converge to the optimal solution.

The semi-gradient method guarantees convergence to the optimal policy for any choice of learning rate, while the full gradient method may diverge with certain learning rates.