Figure from the original question paper

(A) Option figure from the original question paper

(B) Option figure from the original question paper

(C) Option figure from the original question paper

(D) Option figure from the original question paper

(E) Option figure from the original question paper

September 2023 term · Reinforcement Learning · BSDA5007

Reinforcement Learning Quiz 2: 3 December 2023 (September 2023 term)

The IIT Madras BS Reinforcement Learning (Reinforcement Learning) Quiz 2 paper sat on 3 Dec 2023, in the September 2023 term: 21 questions for 50 marks in 120 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.

Questions: 21
Marks: 50
Duration: 120 min
Numerical: 7
MCQ: 9
MSQ: 5

Take as mock test Practise with answers

Updated 25 Sept 2026

Official paper: IIT M DEGREE AN2 EXAM QDB2 03 Dec 2023 · No negative marking.

Question 1

+2 marksNumerical answer

Note: For numerical answer type questions, always enter your answer correct upto two decimal places without rounding up or off.

Show answer

Correct answer: 10

Question 1 on its own page

Question 2

+2 marksNumerical answer

Based on the above data, answer the given subquestions.

What is the estimate of V (B) returned by first-visit MC?
Note: For numerical answer type questions, always enter your answer correct upto two decimal places without rounding up or off.

Show answer

Correct answer: 0.5 (accepted within ±0.05)

Question 2 on its own page

Question 3

+3 marksNumerical answer

Based on the above data, answer the given subquestions.

What is the estimate of V (A) returned by first-visit MC?
Note: For numerical answer type questions, always enter your answer correct upto two decimal places without rounding up or off.

Show answer

Correct answer: 0.975 (accepted within ±0.005)

Question 3 on its own page

Question 4

+2 marksNumerical answer

Based on the above data, answer the given subquestions.

What is the estimate of V (B) returned by batch TD(0)?
Note: For numerical answer type questions, always enter your answer correct upto two decimal places without rounding up or off.

Show answer

Correct answer: 0.5 (accepted within ±0.05)

Question 4 on its own page

Question 5

+3 marksNumerical answer

Based on the above data, answer the given subquestions.

What is the estimate of V(A) returned by batch TD(0)?
Note: For numerical answer type questions, always enter your answer correct upto two decimal places without rounding up or off.

Show answer

Correct answer: 1.2 (accepted within ±0.05)

Question 5 on its own page

Question 6

+2 marksOne correct option

Show answer

Correct answer

Question 6 on its own page

Question 7

+2 marksOne correct option

Which of the following is correct update rule for expected SARSA:

Show answer

Correct answer

Question 7 on its own page

Question 8

+2 marksOne correct option

Choose the correct option:

A
SARSA will have more maximization bias compared to Q-learning.
B
Q-learning will have more maximization bias compared to SARSA.
C
SARSA and Q-learning will have nearly the same maximization bias.
D
Maximization bias for SARSA and Q-learning can not be compared.
E
None of these.

Show answer

Correct answer

B
Q-learning will have more maximization bias compared to SARSA.

Question 8 on its own page

Question 9

+3 marksOne correct option

Consider the following and select the correct option.
Assertion: Double Q learning removes or minimizes maximization bias.
Reason: The target action for Q1 is chosen from Q2 estimate and vice versa.

A
Assertion and Reason are both true and Reason is a correct explanation of Assertion
B
Assertion and Reason are both true and Reason is not a correct explanation of Assertion
C
Assertion is true and Reason is false
D
Both Assertion and Reason are false

Show answer

Correct answer

C
Assertion is true and Reason is false

Question 9 on its own page

Question 10

+3 marksOne correct option

Identify the correct Q function estimated by Dueling DQN?

Show answer

Correct answer

Question 10 on its own page

Question 11

+2 marksOne or more correct options

What is the advantage of computing n^(th) step reward vs 1-step reward?

Select all that apply.

Show answer

Correct answers

Question 11 on its own page

Question 12

+3 marksOne or more correct options

Select correct statements regarding eligibility traces:

Select all that apply.

A
It is an approach to implement the T D(λ) algorithm.
B
The state closest to the terminal state in an episode, has the least value of eligibility trace.
C
It is initialized as 0 before every trajectory in the TD algorithm.
D
It is initialized as 0 at the beginning of the TD algorithm.
E
None of these.

Show answer

Correct answers

A
It is an approach to implement the T D(λ) algorithm.
C
It is initialized as 0 before every trajectory in the TD algorithm.

Question 12 on its own page

Question 13

+3 marksOne correct option

Based on the above data, answer the given subquestions.

What are the Q values for all the actions (i.e. left, up, right, down in that order) in state s1?

A
[2, 5, 8, 11]
B
[2, 5, 11, 8]
C
[5, 2, 8, 11]
D
[11, 8, 2, 5]
E
None of these.

Show answer

Correct answer

A
[2, 5, 8, 11]

Question 13 on its own page

Question 14

+3 marksNumerical answer

Based on the above data, answer the given subquestions.

What is the TD(0) target (Q-learning) for this transition?
Note: For numerical answer type questions, always enter your answer correct upto two decimal places without rounding up or off.

Show answer

Correct answer: 17

Question 14 on its own page

Question 15

+3 marksNumerical answer

Based on the above data, answer the given subquestions.

Note: For numerical answer type questions, always enter your answer correct upto two decimal places without rounding up or off.

Show answer

Correct answer: 8

Question 15 on its own page

Question 16

+2 marksOne or more correct options

Training a neural network with TD target, can diverge because:

Select all that apply.

A
High correlation between consecutive samples.
B
Target keeps moving as fast as the function approximator.
C
The target is moved towards the function approximator.
D
The features are learnt from scratch.
E
None of these.

Show answer

Correct answers

A
High correlation between consecutive samples.
B
Target keeps moving as fast as the function approximator.

Question 16 on its own page

Question 17

+2 marksOne or more correct options

Suppose we are using a policy gradient method to solve a reinforcement learning problem. Assuming that the policy returned by the method is not optimal, which among the following are the plausible reasons for such an outcome?

Select all that apply.

A
The search procedure converged to a locally optimal policy.
B
The search procedure was terminated before it could reach an optimal policy.
C
An optimal policy could not be represented by the parameterisation used to represent the policy.
D
None of these

Show answer

Correct answers

A
The search procedure converged to a locally optimal policy.
B
The search procedure was terminated before it could reach an optimal policy.
C
An optimal policy could not be represented by the parameterisation used to represent the policy.

Question 17 on its own page

Question 18

+2 marksOne or more correct options

What are the advantages of policy search methods over other approaches?

Select all that apply.

A
They can lead to simpler solution description.
B
They offer better convergence as compared to function approximation based methods.
C
In continuous action setting, they work better than value function based approaches.
D
They are robust to partial observability.
E
None of these.

Show answer

Correct answers

A
They can lead to simpler solution description.
B
They offer better convergence as compared to function approximation based methods.
C
In continuous action setting, they work better than value function based approaches.
D
They are robust to partial observability.