The IIT Madras BS Reinforcement Learning (Reinforcement Learning) End Term paper sat on 1 Sept 2024, in the May 2024 term, set QDB3: 22 questions for 50 marks in 180 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.
Consider following assertion reason pair:
Assertion: Reinforcement learning is a type of unsupervised learning algorithm as both don’t have correct labels.
Reason: In unsupervised learning, a reward like quantity is not maximized.
Assertion and Reason are both true and Reason is a correct explanation of Assertion.
Assertion and Reason are both true and Reason is not a correct explanation of Assertion.
Assertion is true but Reason is false.
Assertion is false but Reason is true.
Correct answer
Assertion is false but Reason is true.
Which of these statements is true regarding the rewards obtained in an MDP?
Correct answer
Consider a reinforcement learning agent trying to balance a pole in a continuous environment. The agent receives a reward of +1 for each time step the pole remains balanced and 0 otherwise. Which of the following statements accurately describes the differences between Monte Carlo and Temporal Difference (TD) learning in this scenario?
Monte Carlo methods update the value function based on complete episodes, while TD methods update after each step.
TD methods are guaranteed to converge to the optimal policy, while Monte Carlo methods may not converge.
Monte Carlo methods are less sensitive to the choice of the discount factor compared to TD methods.
TD methods are more effective in environments with high variance and stochasticity compared to Monte Carlo methods.
None of these.
Correct answer
Monte Carlo methods update the value function based on complete episodes, while TD methods update after each step.
Consider a reinforcement learning agent learning to control a robotic arm in a simulated environment. The agent receives a reward of +1 for successfully placing an object in target location and 0 otherwise. Which of the following statements accurately describes a difference between SARSA and Q-learning in this scenario?
SARSA updates its action-value function using the action taken in the next state, while Q-learning updates its action-value function using the maximum action-value across all possible actions in the next state.
SARSA is less sensitive to the choice of policy compared to Q-learning, making it more robust in environments with frequent changes.
SARSA converges more quickly than Q-learning in environments with high reward variance.
Q-learning is inherently more computationally efficient than SARSA due to fewer updates required per episode.
None of these.
Correct answer
SARSA updates its action-value function using the action taken in the next state, while Q-learning updates its action-value function using the maximum action-value across all possible actions in the next state.
In Q-learning, what is the impact of maximization bias on the algorithm’s performance, especially in environments with noisy or stochastic rewards?
Maximization bias can lead to an overestimation of action values, causing the algorithm to favor suboptimal actions and potentially delaying convergence to the optimal policy.
Maximization bias tends to enhance the algorithm’s performance by consistently selecting actions with higher estimated values, leading to quicker convergence.
Maximization bias may result in excessive exploration, allowing the algorithm to discover better strategies in environments with highly variable rewards.
Maximization bias typically has a minor effect on the performance of Q- learning, as the exploration process naturally mitigates its influence over time.
None of these.
Correct answer
Maximization bias can lead to an overestimation of action values, causing the algorithm to favor suboptimal actions and potentially delaying convergence to the optimal policy.
Correct answer
Correct answer
What key issue in standard DQN does the dueling architecture seek to address?
The problem of ensuring faster convergence in environments with high- dimensional state spaces.
The challenge of distinguishing between actions in environments with a large number of potential moves.
The inefficiency in accurately evaluating actions that do not significantly affect the overall value of the state.
The difficulty in learning effective value functions when rewards are sparse and infrequent.
None of these.
Correct answer
The inefficiency in accurately evaluating actions that do not significantly affect the overall value of the state.
Which of the following best captures the key difference between policy gradient methods and Q- learning in reinforcement learning?
Policy gradient methods directly optimize the policy by computing gradients of the expected reward, while Q-learning indirectly improves the policy by learning a value function that estimates the expected rewards for actions.
Q-learning focuses on optimizing the policy parameters directly, while policy gradient methods estimate the action-value function to guide policy improvement.
Policy gradient methods require a model of the environment’s dynamics to compute gradients, while Q-learning does not rely on any such model.
Q-learning is used primarily for continuous action spaces, while policy gradient methods are better suited for discrete action spaces.
None of these.
Correct answer
Policy gradient methods directly optimize the policy by computing gradients of the expected reward, while Q-learning indirectly improves the policy by learning a value function that estimates the expected rewards for actions.
Consider following assertion reason pair:
Assertion: A3C can provide better performance compared to A2C, provided the updates are small enough.
Reason: In A2C, if a thread runs for a long time, other threads have to wait for it to finish.
Assertion and Reason are both true and Reason is a correct explanation of Assertion.
Assertion and Reason are both true and Reason is not a correct explanation of Assertion.
Assertion is true but Reason is false.
Assertion is false but Reason is true.
Correct answer
Assertion and Reason are both true and Reason is a correct explanation of Assertion.
What role do ”meta-policies” play in Hierarchical Reinforcement Learning?
They define the specific actions to take in each state.
They manage the high-level decisions and selection of subtasks.
They directly compute the rewards for each subtask.
They optimize the low-level action policies.
None of these.
Correct answer
They manage the high-level decisions and selection of subtasks.
In HRL, what is a ”subtask” typically used for?
To evaluate the performance of the overall policy
To execute the final decision made by the high-level policy
To break down complex tasks into more manageable components
To directly interact with the environment and collect rewards
None of these.
Correct answer
To break down complex tasks into more manageable components
How does HRL handle long-term dependencies in tasks?
By using recurrent neural networks (RNNs)
By focusing on immediate rewards only
By leveraging hierarchical structures to manage dependencies
By reducing the state space through dimensionality reduction
None of these.
Correct answer
By leveraging hierarchical structures to manage dependencies
What is a common challenge when implementing HRL in practice?
Finding suitable reward functions for all levels
Scaling the approach to very large state spaces
Ensuring the subtasks are independent of each other
Integrating HRL with existing non-hierarchical methods
None of these.
Correct answer
Finding suitable reward functions for all levels
Choose the correct formula for UCT:
Correct answer
Correct answers
Select the correct statements about Generalized Policy Iteration (GPI).
GPI lets policy evaluation and policy improvement interact with each other regardless of the details of the two processes.
At the end of evaluation, the policy is not greedy with respect to the value function computed
GPI converges only when a policy has been found which is greedy with respect to its own value function.
The policy and value function found by GPI at convergence will both be optimal.
None of these.
Correct answers
GPI lets policy evaluation and policy improvement interact with each other regardless of the details of the two processes.
At the end of evaluation, the policy is not greedy with respect to the value function computed
GPI converges only when a policy has been found which is greedy with respect to its own value function.
The policy and value function found by GPI at convergence will both be optimal.
Correct answer: 10.115 (accepted within ±0.185)
Correct answer: 1.4 (accepted within ±0.05)
Correct answer: 2.9 (accepted within ±0.05)
Correct answer: 4.35 (accepted within ±0.05)
Correct answer: 8 (accepted within ±0.05)
