uiz Space

September 2025 term · Reinforcement Learning · BSDA5007

Reinforcement Learning Quiz 2: 23 November 2025 (September 2025 term)

The IIT Madras BS Reinforcement Learning (Reinforcement Learning) Quiz 2 paper sat on 23 Nov 2025, in the September 2025 term: 17 questions for 50 marks in 120 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.

Questions
17
Marks
50
Duration
120 min
MCQ
8
MSQ
5
Numerical
4
Take as mock testPractise with answers

Updated

Official paper: IIT M DEGREE AN EXAM QDB2 23 Nov 2025 NEW · No negative marking.

Question 1

+2 marksOne correct option

Consider the following assertion reason pair:
Assertion: In Monte Carlo (MC), the value function converges to the certainty equivalence estimate.
Reason: Monte Carlo methods use complete trajectories to fit the value function as closely as possible to the sampled returns.

  1. A

    Both Assertion and Reason are correct, and Reason is the correct explanation.

  2. B

    Assertion is correct, Reason is incorrect

  3. C

    Assertion is incorrect, Reason is correct

  4. D

    Both Assertion and Reason are correct, but Reason is not the correct explanation.

Show answer

Correct answer

  • C

    Assertion is incorrect, Reason is correct

Question 1 on its own page

Question 2

+3 marksOne or more correct options

Which of the following is the correct way to represent policy for policy search methods? Assume represents a real-valued parameter.

Select all that apply.

  1. A
  2. B
  3. C
  4. D
  5. E
Show answer

Correct answer

  • A
Question 2 on its own page

Question 3

+3 marksOne or more correct options

Recall the incremental update rule for REINFORCE:

Consider the following binary-bandit problem:

Which of the following expressions are equivalent to
?

Select all that apply.

  1. A
  2. B
  3. C
  4. D
Show answer

Correct answers

  • B
  • C
Question 3 on its own page

Question 4

+2 marksOne or more correct options

In Q-Learning, the update rule for the action-value function is based on bootstrapping from the current estimate. Which of the following correctly represents this update?

Select all that apply.

  1. A
  2. B
  3. C
  4. D
Show answer

Correct answers

  • A
  • D
Question 4 on its own page

Question 5

+3 marksNumerical answer

Consider an episodic task where the
-step returns follow a decaying exponential pattern:

Assume the episode is sufficiently long so the forward-view infinite sum is valid.

Compute the
-return for
. Enter your answer correct to two decimal places.

Show answer

Correct answer: 7.14 (accepted within ±0.01)

Question 5 on its own page

Question 6

+2 marksOne correct option

In Deep Q-Networks (DQN), a separate target network is maintained, whose weights are periodically copied from the main (online) network instead of being updated at every step. Based on the above data, answer the given subquestions.

What is the primary reason for using a separate target network with periodically updated weights in DQN?

  1. A

    To accelerate convergence by increasing the learning rate.

  2. B

    To stabilize learning by keeping target values fixed for several steps.

  3. C

    To ensure the Q-values are always up to date.

  4. D

    To share weights between actor and critic models.

Show answer

Correct answer

  • B

    To stabilize learning by keeping target values fixed for several steps.

Question 6 on its own page

Question 7

+2 marksOne or more correct options

In Deep Q-Networks (DQN), a separate target network is maintained, whose weights are periodically copied from the main (online) network instead of being updated at every step. Based on the above data, answer the given subquestions.

What are the effects of periodically updating the weights of the target network in DQN?

Select all that apply.

  1. A

    Provides stable targets to the main network during training.

  2. B

    Causes the learning targets to be non-stationary.

  3. C

    Mitigates instabilities in Q-learning updates.

  4. D

    Avoids the need for a replay buffer.

Show answer

Correct answers

  • A

    Provides stable targets to the main network during training.

  • C

    Mitigates instabilities in Q-learning updates.

Question 7 on its own page

Question 8

+1 markOne correct option

Based on the above data, answer the given subquestions.

Find the Q values for all three actions
for the state
. Express your answer as a vector q,

where

  1. A
  2. B
  3. C
  4. D
Show answer

Correct answer

  • A
Question 8 on its own page

Question 9

+4 marksNumerical answer

Based on the above data, answer the given subquestions.

Compute the TD target for this transition using SARSA with
. Use a greedy policy derived

from the Q-values at
to select the next action.

Show answer

Correct answer: 0

Question 9 on its own page

Question 10

+4 marksOne correct option

Based on the above data, answer the given subquestions.

Perform one Step of semi-gradient TD using this transition. What would be
if
?

  1. A
  2. B
  3. C
  4. D
Show answer

Correct answer

  • B
Question 10 on its own page

Question 11

+4 marksOne correct option

Consider the following algorithm for learning action-value estimates in an episodic MDP:

Based on the above data, answer the given subquestions.

Which of the following issues in standard Q-learning does the given algorithm aim to reduce?

  1. A

    Overestimation of action values due to maximisation bias.

  2. B

    Underestimation of terminal state values.

  3. C

    Instability due to non-stationary rewards.

  4. D

    Bias introduced by
    -greedy exploration.

Show answer

Correct answer

  • A

    Overestimation of action values due to maximisation bias.

Question 11 on its own page

Question 12

+4 marksOne or more correct options

Consider the following algorithm for learning action-value estimates in an episodic MDP:

Based on the above data, answer the given subquestions.

In the given pseudocode, the algorithm maintains two value functions,
and
, and updates them alternately.
Which of the following best describes the purpose and effect of this design choice?

Select all that apply.

  1. A

    It ensures that the same target network is used for both selection and evaluation, improving stability.

  2. B

    It ensures the overall estimate equals the true expected return by averaging

    and
    .

  3. C

    It decorrelates the selection and evaluation of the greedy action, reducing overestimation bias.

  4. D

    It doubles the effective learning rate by updating two estimators in parallel.

Show answer

Correct answer

  • C

    It decorrelates the selection and evaluation of the greedy action, reducing overestimation bias.

Question 12 on its own page

Question 13

+3 marksOne correct option

Recall from the lectures, the forward-view
-return is defined as:

where
denotes the
-step return starting at time
.Consider using the TD(
) algorithm for an episodic task.
Based on the above data, answer the given subquestions.

What is the effect of setting
in TD(
) on the weighting of
-step returns?

  1. A

    All
    -step returns are ignored.

  2. B

    Only the 1-step return contributes, with weight 1.

  3. C

    All
    -step returns contribute equally.

  4. D

    The weighting distribution remains unchanged.

Show answer

Correct answer

  • B

    Only the 1-step return contributes, with weight 1.

Question 13 on its own page

Question 14

+3 marksOne correct option

Recall from the lectures, the forward-view
-return is defined as:

where
denotes the
-step return starting at time
.Consider using the TD(
) algorithm for an episodic task.
Based on the above data, answer the given subquestions.

As
in an episodic TD(
) task, the
-return
approaches which of the following?

  1. A

    The Monte Carlo return.

  2. B

    The one-step TD target.

  3. C

    The average of all
    -step returns.

  4. D

    Zero, since weights vanish as
    .

Show answer

Correct answer

  • A

    The Monte Carlo return.

Question 14 on its own page

Question 15

+3 marksNumerical answer

Consider an episodic task with four non-terminal states: A, B, C and D. The following are some episodes experienced by an agent following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes.
A,1, C,1, D,0, C,0, A,1, B,1
A,0, C,0, D,1, C,1, B,1
D,1, C,0, A,0, C,1, B,0
C,0, A,1, C,1, B,0
C,1, B,0
D,0, B,1
B,1
B,0
(Enter your answer correct to three decimal places.)
Based on the above data, answer the given subquestions.

Using every-visit Monte Carlo, estimate
and
. What is the value of
?

(use
)

Show answer

Correct answer: 1.975 (accepted within ±0.005)

Question 15 on its own page

Question 16

+5 marksNumerical answer

Consider an episodic task with four non-terminal states: A, B, C and D. The following are some episodes experienced by an agent following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes.
A,1, C,1, D,0, C,0, A,1, B,1
A,0, C,0, D,1, C,1, B,1
D,1, C,0, A,0, C,1, B,0
C,0, A,1, C,1, B,0
C,1, B,0
D,0, B,1
B,1
B,0
(Enter your answer correct to three decimal places.)
Based on the above data, answer the given subquestions.

Using every-visit Monte Carlo, estimate
and
. What is the value of
? (use

)

Show answer

Correct answer: 4.17 (accepted within ±0.01)

Question 16 on its own page

Question 17

+2 marksOne correct option

Consider an episodic task with four non-terminal states: A, B, C and D. The following are some episodes experienced by an agent following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes.
A,1, C,1, D,0, C,0, A,1, B,1
A,0, C,0, D,1, C,1, B,1
D,1, C,0, A,0, C,1, B,0
C,0, A,1, C,1, B,0
C,1, B,0
D,0, B,1
B,1
B,0
(Enter your answer correct to three decimal places.)
Based on the above data, answer the given subquestions.

Batch Monte Carlo (MC) methods estimate the value function by finding the least-squares fit to the sampled returns generated under the policy.

  1. A

    TRUE

  2. B

    FALSE

Show answer

Correct answer

  • A

    TRUE

Question 17 on its own page