Question text from the original paper, with its maths as pictures

(B) Initiation set, option policy, termination function

(A) State space, action space, reward function

(C) Value function, behaviour policy, discount factor

(D) Transition model, reward model, termination condition

Question text from the original paper, with its maths as pictures

(B) To stabilise training by preventing the TD target from changing too rapidly

(A) To prevent the actor from updating faster than the critic

(C) To ensure the replay buffer contains on-policy data

(D) To match the learning rate of the actor and critic networks

Question text from the original paper, with its maths as pictures Figure from the original question paper Question text from the original paper, with its maths as pictures

(B) Question text from the original paper, with its maths as pictures

(A) Question text from the original paper, with its maths as pictures

(C) Question text from the original paper, with its maths as pictures

(D) The surrogate eliminates the need for a value function baseline

Question text from the original paper, with its maths as pictures

(D) Question text from the original paper, with its maths as pictures

(A) Question text from the original paper, with its maths as pictures

(B) Question text from the original paper, with its maths as pictures

(C) Question text from the original paper, with its maths as pictures

Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Why is DPG generally more sample-efficient than SPG in high-dimensional continuous action spaces?

(B) Question text from the original paper, with its maths as pictures

(A) Question text from the original paper, with its maths as pictures

(C) DPG uses a target network which reduces gradient variance automatically

(D) DPG enforces determinism, which eliminates exploration noise and thus requires fewer environment interactions

Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Ganesh’s on-policy DPG implementation is likely to suffer from which of the following problems? Select all that apply.

(A) Question text from the original paper, with its maths as pictures

(C) The DPG theorem technically requires an off-policy behaviour policy to ensure sufficient state coverage; on-policy data violates this

(D) Question text from the original paper, with its maths as pictures

(B) Question text from the original paper, with its maths as pictures

(E) Question text from the original paper, with its maths as pictures

Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Question text from the original paper, with its maths as pictures

(A) Question text from the original paper, with its maths as pictures

(B) Question text from the original paper, with its maths as pictures

(C) Question text from the original paper, with its maths as pictures

(D) Question text from the original paper, with its maths as pictures

(E) Question text from the original paper, with its maths as pictures

Question text from the original paper, with its maths as pictures Figure from the passage in the original paper The REINFORCE gradient estimator is: Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Why does REINFORCE exhibit high gradient variance, particularly in long-horizon tasks like text generation?

(B) Question text from the original paper, with its maths as pictures

(A) Question text from the original paper, with its maths as pictures

(C) Monte Carlo rollouts are biased estimators of the true return

(D) Question text from the original paper, with its maths as pictures

Question text from the original paper, with its maths as pictures Figure from the passage in the original paper The REINFORCE gradient estimator is: Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Question text from the original paper, with its maths as pictures

(A) Question text from the original paper, with its maths as pictures

(B) Question text from the original paper, with its maths as pictures

(C) Question text from the original paper, with its maths as pictures

(D) Question text from the original paper, with its maths as pictures

Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures Figure from the passage in the original paper Question text from the original paper, with its maths as pictures What is the key advantage of Intra-Option Q-Learning (Algorithm B) over SMDP Q-Learning (Algorithm A)?

(B) Question text from the original paper, with its maths as pictures

(A) Question text from the original paper, with its maths as pictures

(C) It converges to a globally optimal policy, whereas SMDP Q-learning only converges locally

(D) It eliminates the need for a high-level policy by learning all options simultaneously

January 2026 term · Reinforcement Learning · BSDA5007

Reinforcement Learning End Term: 10 May 2026, Set 1 (January 2026 term)

The IIT Madras BS Reinforcement Learning (Reinforcement Learning) End Term paper sat on 10 May 2026, in the January 2026 term, set 1: 20 questions for 30 marks in 180 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.

Questions: 20
Marks: 30
Duration: 180 min
MCQ: 7
MSQ: 7
Numerical: 6

Take as mock test Practise with answers

Updated 25 Sept 2026

Official paper: Reinforcement Learning 10 May 26 (Session 2) · No negative marking.

Question 1

+1 markOne correct option

A
State space, action space, reward function
B
Initiation set, option policy, termination function
C
Value function, behaviour policy, discount factor
D
Transition model, reward model, termination condition

Show answer

Correct answer

B
Initiation set, option policy, termination function

Question 1 on its own page

Question 2

+1 markOne correct option

A
To prevent the actor from updating faster than the critic
B
To stabilise training by preventing the TD target from changing too rapidly
C
To ensure the replay buffer contains on-policy data
D
To match the learning rate of the actor and critic networks

Show answer

Correct answer

B
To stabilise training by preventing the TD target from changing too rapidly

Question 2 on its own page

Question 3

+1 markOne correct option

DDPG uses an experience replay buffer. What is the direct consequence of using an experience replay buffer

A
The policy gradient estimate becomes unbiased
B
The algorithm becomes on-policy
C
Temporal correlations between consecutive samples are broken
D
The critic no longer requires a target network

Show answer

Correct answer

C
Temporal correlations between consecutive samples are broken

Question 3 on its own page

Question 4

+1 markOne correct option

Compared to the policy gradient, the DPG theorem requires integration over:

A
Both state and action spaces
B
State space only
C
Action space only
D
Neither — it requires only a single action sample per state

Show answer

Correct answer

B
State space only

Question 4 on its own page

Question 5

+0 marksOne correct option

A
B
C
D
The surrogate eliminates the need for a value function baseline

Show answer

Correct answer

Question 5 on its own page

Question 6

+1 markOne or more correct options

Which are known limitations or failure modes of DDPG?

Select all that apply.

A
High sensitivity to hyperparameters such as learning rate and architecture
B
Inability to handle continuous action spaces
C
Brittle training that may diverge without careful tuning
D
Deterministic policy requires explicit exploration noise during training

Show answer

Correct answers

A
High sensitivity to hyperparameters such as learning rate and architecture
C
Brittle training that may diverge without careful tuning
D
Deterministic policy requires explicit exploration noise during training

Question 6 on its own page

Question 7

+2 marksOne or more correct options

Select all that apply.

Show answer

Correct answer

Question 7 on its own page

Question 8

+3 marksOne or more correct options

Recall standard control algorithms in RL, like Q-Learning, SARSA, Expected SARSA, etc., involve a maximisation step in constructing their target policies.
For example: In Q-Learning, the target is:

Which of the following statements are correct? Select all that apply.

Select all that apply.

A
(A figure from the original paper is missing from the source site.)
B
(A figure from the original paper is missing from the source site.)
C
D
(A figure from the original paper is missing from the source site.)
E
F

Show answer

Correct answers

B
(A figure from the original paper is missing from the source site.)
C
E
F

Question 8 on its own page

Question 9

+1 markNumerical answer

Show answer

Correct answer: 3.615 (accepted within ±0.015)

Question 9 on its own page

Question 10

+0 marksNumerical answer

Round your final answer to 2 decimal places.

Show answer

Correct answer: 3.24 (accepted within ±0.02)

Question 10 on its own page

Question 11

+2 marksOne correct option

Why is DPG generally more sample-efficient than SPG in high-dimensional continuous action spaces?

A
B
C
DPG uses a target network which reduces gradient variance automatically
D
DPG enforces determinism, which eliminates exploration noise and thus requires fewer environment interactions

Show answer

Correct answer

Question 11 on its own page

Question 12

+2 marksOne or more correct options

Ganesh’s on-policy DPG implementation is likely to suffer from which of the following problems? Select all that apply.

Select all that apply.

A
B
C
The DPG theorem technically requires an off-policy behaviour policy to ensure sufficient state coverage; on-policy data violates this
D
E

Show answer

Correct answers

A
C
The DPG theorem technically requires an off-policy behaviour policy to ensure sufficient state coverage; on-policy data violates this
D

Question 12 on its own page

Question 13

+0 marksNumerical answer

Show answer

Correct answer: 0.0215 (accepted within ±0.0005)

Question 13 on its own page

Question 14

+0 marksOne or more correct options

Select all that apply.

Show answer

Correct answers

Question 14 on its own page

Question 15

+2 marksOne or more correct options

The REINFORCE gradient estimator is:

Why does REINFORCE exhibit high gradient variance, particularly in long-horizon tasks like text generation?

Select all that apply.

A
B
C
Monte Carlo rollouts are biased estimators of the true return
D

Show answer

Correct answer

Question 15 on its own page

Question 16

+2 marksNumerical answer

The REINFORCE gradient estimator is:

Show answer

Correct answer: 0.00675 (accepted within ±0.00005)

Question 16 on its own page

Question 17

+2 marksOne or more correct options

The REINFORCE gradient estimator is:

Select all that apply.

Show answer

Correct answer

Question 17 on its own page

Question 18

+2 marksOne correct option

What is the key advantage of Intra-Option Q-Learning (Algorithm B) over SMDP Q-Learning (Algorithm A)?

A
B
C
It converges to a globally optimal policy, whereas SMDP Q-learning only converges locally
D
It eliminates the need for a high-level policy by learning all options simultaneously

Show answer

Correct answer

Question 18 on its own page

Question 19

+2 marksNumerical answer

Show answer

Correct answer: 1.1 (accepted within ±0.05)

Question 19 on its own page

Question 20

+2 marksNumerical answer

Show answer

Correct answer: 3.11 (accepted within ±0.05)

Question 20 on its own page