The IIT Madras BS Reinforcement Learning (Reinforcement Learning) End Term paper sat on 13 Apr 2025, in the January 2025 term, set 1-2: 16 questions for 43 marks in 180 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.
In which of the following scenarios is Expected SARSA a good fit?
When the environment has a high degree of stochasticity, makingbootstrapping unstable in value-based methods
When function approximation is necessary due to large state spaces, requiringdeep networks for learning representations
When learning needs to prioritize separating state value and advantagefunctions for better decision-making
When experience replay is essential for stable learning and sample efficiency
Correct answer
When the environment has a high degree of stochasticity, makingbootstrapping unstable in value-based methods
SARSA updates its action-value function based on the action actually taken inthe next state, while Q-learning updates its action-value function based on the maximum action-value in the next state.
SARSA is guaranteed to converge to the optimal policy under certainconditions, while Q-learning may diverge or oscillate without additional modifications.
SARSA is more computationally efficient than Q-learning, requiring fewerupdates to converge to the optimal policy.
SARSA and Q-learning exhibit similar performance in terms of convergencespeed and solution quality in this scenario.
Correct answer
SARSA updates its action-value function based on the action actually taken inthe next state, while Q-learning updates its action-value function based on the maximum action-value in the next state.
Choose the correct statement in context of multi armed bandits (MAB), assuming stationary and normal reward distribution:
If there are n arms, the lower bound of finding the optimal arm is ω(n).
If an agent doesn’t sufficiently pull each arm, then it can incorrectly pick asuboptimal arm as the optimal arm.
Exploration is very important step and an agent should keep exploringregularly, to minimize the regret.
The optimal arm can be determined by pulling each arm once and then itshould be pulled every time afterwards.
Correct answers
If there are n arms, the lower bound of finding the optimal arm is ω(n).
If an agent doesn’t sufficiently pull each arm, then it can incorrectly pick asuboptimal arm as the optimal arm.
Which of the following methods are a form of Generalized Policy Iteration?
Policy Improvement
Value Iteration
Q-Learning and SARSA
Monte Carlo control methods
Correct answers
Value Iteration
Q-Learning and SARSA
Monte Carlo control methods
b can be a function of s
b can be a function of a
b can be a random variable independent of s, a
None of these
Correct answers
b can be a function of s
b can be a random variable independent of s, a
It increases the variance in the estimate of the gradient of the performance.
It decreases the variance in the estimate of the gradient of the performance.
It introduces a bias in the estimate of the gradient of the performance.
It doesn’t introduce any bias in the estimate of the gradient of theperformance.
None of these.
Correct answers
It decreases the variance in the estimate of the gradient of the performance.
It introduces a bias in the estimate of the gradient of the performance.
Correct answer: -6
Based on the above data, answer the given subquestions.
Correct answer: 14
Based on the above data, answer the given subquestions.
Correct answer: -5
Based on the above data, answer the given subquestions.
Correct answer: 40
Based on the above data, answer the given subquestions.
Correct answer: 0.96 (accepted within ±0.01)
Based on the above data, answer the given subquestions.
Which of the routes illustrated on the grid is taken when the Hierarchically Optimal policy is executed?
Route 1
Route 2
Route 3
Correct answer
Route 2
Based on the above data, answer the given subquestions.
Which of the routes illustrated on the grid is taken when the Flat Optimal policy is executed?
Route 1
Route 2
Route 3
Correct answer
Route 3
Based on the above data, answer the given subquestions.
Calculate the number of time-steps taken to finish the episode when the re-cursively optimal policy is executed.
Correct answer: 12
Based on the above data, answer the given subquestions.
Correct answer: -3.62 (accepted within ±0.01)
Based on the above data, answer the given subquestions.
Which action would you pick, after having performed this rollout?
Action a1
Action a2
Both actions are equally rewarding
Correct answer
Action a1
