uiz Space

January 2026 term · Reinforcement Learning · BSDA5007

Reinforcement Learning End Term: 10 May 2026, Set S2 (January 2026 term)

The IIT Madras BS Reinforcement Learning (Reinforcement Learning) End Term paper sat on 10 May 2026, in the January 2026 term, set S2: 21 questions for 30 marks in 180 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.

Questions
21
Marks
30
Duration
180 min
MCQ
8
MSQ
7
Numerical
6
Take as mock testPractise with answers

Updated

Official paper: Reinforcement Learning 10 May 26 (Session 2) · No negative marking.

Question 1

+1 markOne correct option
  1. A

    The discount factor used during option execution

  2. B
  3. C
  4. D
Show answer

Correct answer

  • C
Question 1 on its own page

Question 2

+1 markOne correct option

In DDPG, exploration is typically achieved by:

  1. A
  2. B

    Randomly sampling from the replay buffer with higher priority

  3. C

    Injecting noise into the critic network parameters

  4. D

    Adding time-correlated noise (e.g., Ornstein-Uhlenbeck) or Gaussian noise to the deterministic policy output

Show answer

Correct answer

  • D

    Adding time-correlated noise (e.g., Ornstein-Uhlenbeck) or Gaussian noise to the deterministic policy output

Question 2 on its own page

Question 3

+1 markOne correct option
  1. A

    Reduces variance without introducing bias

  2. B

    Eliminates all bias in the gradient estimate

  3. C

    Makes the algorithm off-policy

  4. D

    Removes the need for a value function approximator

Show answer

Correct answer

  • A

    Reduces variance without introducing bias

Question 3 on its own page

Question 4

+1 markOne correct option
  1. A
  2. B
  3. C
  4. D
Show answer

Correct answer

  • B
Question 4 on its own page

Question 5

+0 marksOne correct option

[Bonus] TRPO uses a backtracking line search after computing the natural gradient direction. What is the primary purpose of this line search?

  1. A

    To reduce the computational cost of the conjugate gradient solver

  2. B
  3. C

    To transform the natural gradient into a standard gradient

  4. D

    To eliminate the need for importance sampling corrections

Show answer

Correct answer

  • B
Question 5 on its own page

Question 6

+1 markOne or more correct options

Which of the following statements about DDPG are correct?

Select all that apply.

  1. A

    DDPG maintains four networks: actor, critic, target actor, and target critic

  2. B

    The critic is trained by minimising the Bellman error using off-policy data sampled from the replay buffer

  3. C

    DDPG is directly applicable to discrete action spaces without modification

  4. D

    The actor is updated by ascending the gradient of the critic's Q-value with respect to actor parameters

Show answer

Correct answers

  • A

    DDPG maintains four networks: actor, critic, target actor, and target critic

  • B

    The critic is trained by minimising the Bellman error using off-policy data sampled from the replay buffer

  • D

    The actor is updated by ascending the gradient of the critic's Q-value with respect to actor parameters

Question 6 on its own page

Question 7

+2 marksOne or more correct options

Which of the following are not valid conditions required for the DPG theorem to hold?

Select all that apply.

  1. A
  2. B
  3. C

    The policy must be stochastic to ensure sufficient exploration

  4. D

    The MDP transition dynamics must be differentiable

Show answer

Correct answers

  • C

    The policy must be stochastic to ensure sufficient exploration

  • D

    The MDP transition dynamics must be differentiable

Question 7 on its own page

Question 8

+3 marksOne or more correct options

Which of the following statements are correct? Select all that apply.

Select all that apply.

  1. A

    (A figure from the original paper is missing from the source site.)

  2. B

    (A figure from the original paper is missing from the source site.)

  3. C
  4. D

    (A figure from the original paper is missing from the source site.)

  5. E
  6. F
Show answer

Correct answers

  • B

    (A figure from the original paper is missing from the source site.)

  • C
  • E
  • F
Question 8 on its own page

Question 9

+1 markNumerical answer
Show answer

Correct answer: 5.33 (accepted within ±0.02)

Question 9 on its own page

Question 10

+0 marksNumerical answer
Show answer

Correct answer: 3.24 (accepted within ±0.02)

Question 10 on its own page

Question 11

+2 marksOne correct option
  1. A

    It causes the replay buffer to overfit to recent transitions

  2. B
  3. C

    It forces the actor to update faster than the critic, creating a policy-value mismatch

  4. D

    It eliminates the exploration noise injected into actions, causing the policy to get stuck

Show answer

Correct answer

  • B
Question 11 on its own page

Question 12

+2 marksOne or more correct options

Aditi's unstable DDPG implementation is likely to suffer from which of the following problems? Select all that apply.

Select all that apply.

  1. A
  2. B

    Brittle training that may diverge without careful hyperparameter tuning

  3. C
  4. D

    DDPG cannot be applied to 4-dimensional action spaces because of curse of dimensionality in the action space

  5. E

    Temporal correlation between consecutive samples

Show answer

Correct answers

  • A
  • B

    Brittle training that may diverge without careful hyperparameter tuning

  • C
  • E

    Temporal correlation between consecutive samples

Question 12 on its own page

Question 13

+0 marksNumerical answer
Show answer

Correct answer: 0.405 (accepted within ±0.005)

Question 13 on its own page

Question 14

+3 marksOne or more correct options

Which of the following statements about the eligibility vector for a Bernoulli-logistic unit are correct? Select all that apply.

Select all that apply.

  1. A
  2. B
  3. C
  4. D
Show answer

Correct answers

  • A
  • B
  • D
Question 14 on its own page

Question 15

+0 marksOne or more correct options

Select all that apply.

  1. A
  2. B
  3. C
  4. D
  5. E
Show answer

Correct answers

  • A
  • B
  • C
  • D
Question 15 on its own page

Question 16

+2 marksOne correct option

Why does the REINFORCE gradient estimator exhibit high variance, particularly in long-horizon robotic tasks?

  1. A

    The policy network's weights are initialised randomly, causing gradient noise

  2. B
  3. C

    The policy gradient theorem does not hold for tasks with continuous action spaces

  4. D

    REINFORCE requires a critic to reduce variance, which is absent by definition

Show answer

Correct answer

  • B
Question 16 on its own page

Question 17

+2 marksNumerical answer
Show answer

Correct answer: 0.3 (accepted within ±0.05)

Question 17 on its own page

Question 18

+2 marksOne or more correct options

Select all that apply.

  1. A
  2. B
  3. C
  4. D
Show answer

Correct answers

  • A
  • B
  • D
Question 18 on its own page

Question 19

+2 marksOne correct option

Which of the following is a key advantage of Intra-Option Q-Learning over SMDP Q-Learning?

  1. A

    Intra-Option Q-Learning converges to the globally optimal policy, whereas SMDP Q-Learning only converges to locally optimal solutions

  2. B
  3. C
  4. D

    Intra-Option Q-Learning learns only a single option, avoiding the need for a high-level policy

Show answer

Correct answer

  • B
Question 19 on its own page

Question 20

+2 marksNumerical answer
Show answer

Correct answer: 1.65 (accepted within ±0.01)

Question 20 on its own page

Question 21

+2 marksNumerical answer
Show answer

Correct answer: 2.74 (accepted within ±0.01)

Question 21 on its own page