The IIT Madras BS Reinforcement Learning (Reinforcement Learning) Quiz 2 paper sat on 16 Mar 2025, in the January 2025 term: 22 questions for 50 marks in 120 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.
For on policy SARSA update rule for Q(s, a), s, a, s′, a′ are needed. When is a′ computed/sampled?
before updating Q(s, a)
after updating Q(s, a)
before and after updating Q(s, a), but only later value is used.
before and after updating Q(s, a), but only the earlier value is used.
None of these.
Correct answer
before updating Q(s, a)
Correct answer
The eligibility trace is used to store past observations and actions.
The eligibility trace determines the learning rate for updating the value estimates.
The eligibility trace helps in determining which state-action pairs are eligible for updates.
The eligibility trace maintains a history of rewards received in each state.
Correct answer
The eligibility trace helps in determining which state-action pairs are eligible for updates.
What is a key advantage of using n-step TD prediction over one-step TD prediction?
n-step TD prediction requires less memory and computational resources.
n-step TD prediction can handle delayed rewards and credit assignment over multiple time steps.
n-step TD prediction converges faster to the optimal policy.
n-step TD prediction guarantees convergence to the optimal value function for any choice of the learning rate.
Correct answer
n-step TD prediction can handle delayed rewards and credit assignment over multiple time steps.
Consider a reinforcement learning agent navigating a complex environment with sparse rewards. Which of the following statements accurately describes a key advantage of using Double Q- learning in this scenario?
Double Q-learning helps to reduce the variance in action value estimates, leading to more stable learning.
Double Q-learning is guaranteed to converge to the optimal policy even with non-linear function approximation.
Double Q-learning requires fewer samples to achieve the same level of performance compared to traditional Q-learning.
Double Q-learning is less sensitive to hyperparameter tuning, making it easier to train in complex environments.
Correct answer
Double Q-learning helps to reduce the variance in action value estimates, leading to more stable learning.
Expected SARSA
DQN
Double DQN
Dueling DQN
Correct answer
Dueling DQN
Correct answer: 1
Based on the above data, answer the given subquestions.
What is the estimate of V (B) returned by first-visit MC?
Correct answer: 0.75 (accepted within ±0.05)
Based on the above data, answer the given subquestions.
What is the estimate of V (A) returned by first-visit MC?
Correct answer: 1.7 (accepted within ±0.05)
Based on the above data, answer the given subquestions.
What is the estimate of V (B) returned by batch TD(0)?
Correct answer: 0.75 (accepted within ±0.05)
Based on the above data, answer the given subquestions.
What is the estimate of V (A) returned by batch TD(0)?
Correct answer: 2.1 (accepted within ±0.05)
Based on the above data, answer the given subquestions.
V (s) is a linear function approximator.
TRUE
FALSE
Correct answer
TRUE
Based on the above data, answer the given subquestions.
Correct answers
Based on the above data, answer the given subquestions.
Correct answer: -1.2
Based on the above data, answer the given subquestions.
Correct answer: 1.5
Based on the above data, answer the given subquestions.
Correct answers
Based on the above data, answer the given subquestions.
Correct answer
Based on the above data, answer the given subquestions.
The optimal Policy
Uniform Random Policy
Insufficient Information
Correct answer
Uniform Random Policy
Based on the above data, answer the given subquestions.
Yes
No
Correct answer
No
Based on the above data, answer the given subquestions.
What is the value of d (i.e. the length of the discretized state vector)?
Correct answer: 29
Based on the above data, answer the given subquestions.
Correct answer: 3
Based on the above data, answer the given subquestions.
Correct answer: 1
