Reinforcement Learning Quiz 1: 26 October 2025 (September 2025 term)
The IIT Madras BS Reinforcement Learning (Reinforcement Learning) Quiz 1 paper sat on 26 Oct 2025, in the September 2025 term: 16 questions for 50 marks in 120 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.
- 16
- 50
- 120 min
- 11
- 2
- 3
Show answer
Correct answer
Question 2
Consider following assertion reason pair:
Assertion:
In the UCB algorithm for multi-armed bandits, replacing the upper confidence bound with a lower confidence bound (and greedily selecting actions based on that lower bound)
would still promote effective exploration or lead to optimal reward maximisation.
Reason:
Even when lower confidence bounds are used instead of upper bounds, if the algorithm greedily selects arms based on these lower bounds, it would still encourage exploration and ultimately achieve optimal reward maximisation.
Both Assertion and Reason are true, and Reason is the correct explanation of Assertion.
Both Assertion and Reason are true, but Reason is NOT the correct explanation of Assertion.
Assertion is true, Reason is false
Assertion is false, Reason is false
Show answer
Correct answer
Assertion is false, Reason is false
Question 3
Policy Iteration
Value Iteration
Both require equal updates
It cannot be determined
Show answer
Correct answer
Value Iteration
Question 4
Arm 1
Arm 2
Arm 3
Arm 4
Show answer
Correct answer
Arm 3
Question 5
Which of the following statements correctly describes the differences in computational complexity and convergence behaviour between Policy Iteration and Value Iteration algorithms in solving Markov Decision Processes?
Value Iteration requires fewer iterations to converge, but each iteration is computationally more expensive due to the policy evaluation step.
Value Iteration performs a combined update of value estimation and policy improvement in one step and requires more iterations to converge compared to Policy Iteration.
Policy Iteration can only be applied to small state spaces, while Value Iteration scales well to large state spaces.
Value Iteration is simpler to implement because it only maintains a value function, while Policy Iteration maintains both policy and value function.
Both Policy Iteration and Value Iteration are guaranteed to converge to the optimal policy.
Show answer
Correct answers
Value Iteration performs a combined update of value estimation and policy improvement in one step and requires more iterations to converge compared to Policy Iteration.
Value Iteration is simpler to implement because it only maintains a value function, while Policy Iteration maintains both policy and value function.
Both Policy Iteration and Value Iteration are guaranteed to converge to the optimal policy.
Question 6
Show answer
Correct answer: 4
Question 7
Show answer
Correct answer: 0.2
Question 8
Suppose you face a 2-armed bandit task where, at each time step, the true action values are either (10, 20) with probability 0.7 (case A) or (90, 80) with probability 0.3 (case B).
Based on the above data, answer the given subquestions.
If you cannot observe which case you face, what is the best expected reward per step you can achieve, and what strategy should you follow?
Randomly choose between Action 1 and Action 2; expected reward is 3.
Choose Action 1 always; expected reward is 34.
Choose Action 2 always; expected reward is 38.
Alternate between Action 1 and Action 2; expected reward is 40.
Show answer
Correct answer
Choose Action 2 always; expected reward is 38.
Question 9
Suppose you face a 2-armed bandit task where, at each time step, the true action values are either (10, 20) with probability 0.7 (case A) or (90, 80) with probability 0.3 (case B).
Based on the above data, answer the given subquestions.
If, at each time step, you are told whether you face case A or case B (but not the true action values), what is the best expected reward per step you can achieve,and what strategy should you follow?
Pick Action 2 in case A, Action 1 in case B; expected reward is 41.
Always pick Action 2; expected reward is 38.
Always pick Action 1; expected reward is 34.
The expected reward stated in all the given options is incorrect.
Show answer
Correct answer
Pick Action 2 in case A, Action 1 in case B; expected reward is 41.
Question 10
Based on the above data, answer the given subquestions.
Which of the following best explains why a contextual bandit approach is preferred over a standard multi-armed bandit in this scenario?
Because the optimal waiting time is the same for all VMs, regardless of their context.
Because contextual bandits can adaptively select actions based on the specific features of each VM failure event, maximising expected reward across diverse situations.
Because multi-armed bandits are unable to balance exploration and exploitation.
Because contextual bandits always guarantee zero regret after every round.
Show answer
Correct answer
Because contextual bandits can adaptively select actions based on the specific features of each VM failure event, maximising expected reward across diverse situations.
Question 11
Based on the above data, answer the given subquestions.
It determines the penalty for rebooting or migrating the VM.
It is used to predict the expected reward for each possible waiting time,allowing the algorithm to personalize decisions for each failure event.
It is ignored by LinUCB, which only uses past rewards.
It is used to randomly select an action to ensure exploration.
Show answer
Correct answer
It is used to predict the expected reward for each possible waiting time,allowing the algorithm to personalize decisions for each failure event.
Question 12
Based on the above data, answer the given subquestions.
What is the main optimality goal of a contextual bandit algorithm over many rounds?
Maximise the total expected reward by acting as closely as possible to the best policy mapping contexts to actions.
Achieve zero regret after every individual round by always picking the empirically best action so far for each context
Always exploit the highest-reward action seen so far
Minimise the loss on the worst round
Show answer
Correct answer
Maximise the total expected reward by acting as closely as possible to the best policy mapping contexts to actions.
Question 13
Based on the above data, answer the given subquestions.
What is the total reward if the robot collects both treasures and then exits,taking the shortest possible path and never hitting a wall?
Show answer
Correct answer: 76
Question 14
Based on the above data, answer the given subquestions.
Show answer
Correct answers
Question 15
Consider a pole-balancing task where the goal is to apply forces to a cart moving along a track to keep a hinged pole from falling. A failure event occurs when the pole falls past a certain angle or the cart moves off the track, after which the pole resets to vertical.
The problem can be formulated either as an episodic task (with episodes ending at failure) or as a continuing task (after each failure, pole gets back into its initial position,
potentially infinite time horizon with discounting).Rewards can be defined as:
• Episodic: Reward +1 for each timestamp without failure, so the return is the number of steps until failure (possibly infinite if balanced forever).
• Continuing: Reward 0 for each timestamp except at each failure, where the reward is −1. Returns are discounted sums of future rewards.
Based on the above data, answer the given subquestions.
Show answer
Correct answer
Question 16
Consider a pole-balancing task where the goal is to apply forces to a cart moving along a track to keep a hinged pole from falling. A failure event occurs when the pole falls past a certain angle or the cart moves off the track, after which the pole resets to vertical.
The problem can be formulated either as an episodic task (with episodes ending at failure) or as a continuing task (after each failure, pole gets back into its initial position,
potentially infinite time horizon with discounting).Rewards can be defined as:
• Episodic: Reward +1 for each timestamp without failure, so the return is the number of steps until failure (possibly infinite if balanced forever).
• Continuing: Reward 0 for each timestamp except at each failure, where the reward is −1. Returns are discounted sums of future rewards.
Based on the above data, answer the given subquestions.
How does the return in the episodic, discounted pole-balancing task differ from the continuing formulation with the same reward structure?
Episodic return is a single discounted reward at failure; continuing return is an infinite sum over all future failures.
Both returns are always zero
Episodic return accumulates rewards at every time step; continuing return only at failure.
There is no difference between the two formulations.
Show answer
Correct answer
Episodic return is a single discounted reward at failure; continuing return is an infinite sum over all future failures.