Reinforcement Learning Quiz 1: 15 March 2026 (January 2026 term)

Which of the following is/are correct and valid reasons to consider sampling actions from a softmax distribution instead of using an greedy approach? 1.Under softmax exploration, the probability of selecting an action increases with its estimated action value, which reduces unnecessary exploration of clearly inferior actions. 2.Unlike the -greedy method, softmax exploration does not require careful, gradual decay of the exploration parameter and still yields asymptotically correct behaviour even if the temperature is reduced sharply. 3.It enables more fine-grained discrimination among actions whose estimated Q-values are close to the maximum, allowing more nuanced preference for slightly better actions. Which of the above statements is/are correct?

Consider a discounted return:

in an infinite-horizon MDP with bounded rewards and discount factor 1.For fixed rewards, the contribution of to decays geometrically as . 2.If and rewards are uniformly bounded, the infinite sum is always finite. 3.For , the discounted return is bounded above in magnitude by

. Here, Rmax be the maximum reward for any transition. Which of the above statements is/are correct?

Consider the following assertion and reason pair and select the correct option: Assertion:In the UCB algorithm for multi-armed bandits, replacing the upper confidence bound with a lower confidence bound (and greedily selecting actions based on that lower bound) would still promote effective exploration or lead to optimal reward maximisation. Reason:Even when lower confidence bounds are used instead of upper bounds, if the algorithm greedily selects arms based on these lower bounds, it would still encourage exploration and ultimately achieve optimal reward maximisation.

In the context of MDPs, what does it mean for a policy to be greedy with respect to an action- value function ?

In policy iteration for finite MDPs, how are the Bellman equations used during the policy evaluation step?

In a -armed bandit setting, we maintain a running estimate of the action-value for each arm , denoted . We now consider using an upper confidence bound (UCB) style rule for arm selection at time , where is the number of times arm has been selected up to time , and is a tunable hyperparameter. Which of the following are good strategies for arm selection?

Which of the following equations best represents the Bellman optimality equation for the optimal state-value function

The sequence of rewards for a continuing task with is given below:

Find the return . Your answer should have exactly two places after the decimal point.

You have a 3-armed bandit where each arm, when active, yields a Bernoulli reward with success probabilities , , and . However, arm 1 is active in any given round with probability , and otherwise it yields a reward of 0 regardless of its Bernoulli outcome. Arms 2 and 3 are always active. What is the expected reward from pulling arm 1 in a single round?

Consider a Markov Decision Process (MDP) with three states , , and , where is a terminal state and . Each non-terminal state has a single available action, and the transitions are deterministic with associated rewards: the agent moves from to and receives a reward of 1, and from to and receives a reward of 2. Let the discount factor be . Based on the above data, answer the given subquestions.

Assume the initial value estimates are , , and . After performing one synchronous policy evaluation update (under the fixed policy that follows the chain), determine and . Compute the ratio

Consider a Markov Decision Process (MDP) with three states , , and , where is a terminal state and . Each non-terminal state has a single available action, and the transitions are deterministic with associated rewards: the agent moves from to and receives a reward of 1, and from to and receives a reward of 2. Let the discount factor be . Based on the above data, answer the given subquestions.

Starting from , , and . perform two successive synchronous policy evaluation sweeps (under the fixed policy that follows the chain). What is after these two sweeps?

Consider a 2-state MDP with states and , discount factor . There are two actions in : and . Action gives immediate reward and moves deterministically to ; action gives immediate reward and keeps the agent in . In , there is a single action that yields an immediate reward and transitions back to . Assume initial value estimates are , Based on the above data, answer the given subquestions.

If you perform a single value iteration Bellman optimality update for , what is the new value ?

Consider a 2-state MDP with states and , discount factor . There are two actions in : and . Action gives immediate reward and moves deterministically to ; action gives immediate reward and keeps the agent in . In , there is a single action that yields an immediate reward and transitions back to . Assume initial value estimates are , Based on the above data, answer the given subquestions.

Suppose after some value iterations your current estimates are , . If you perform one value iteration update for , what is the new value ?

Consider an N-armed bandit problem in which each arm , for , produces a reward of 1 with probability and otherwise, where denotes the arm’s base success probability. Assume the bandit machine suffers from a hardware malfunction: when the agent attempts to pull arm , the intended action is not always executed. Instead, with probability a wiring fault causes the machine to randomly activate one arm chosen uniformly from the set , regardless of the agent’s selection, and the reward is generated according to that arm’s reward distribution. We refer to this malfunction as -noise activation Based on the above data, answer the given subquestions.

Consider a faulty -armed bandit with -noise activation and base arm success probabilities . If you always choose the arm with the highest base success probability , what is the expected payoff per pull under this greedy strategy?

Consider an N-armed bandit problem in which each arm , for , produces a reward of 1 with probability and otherwise, where denotes the arm’s base success probability. Assume the bandit machine suffers from a hardware malfunction: when the agent attempts to pull arm , the intended action is not always executed. Instead, with probability a wiring fault causes the machine to randomly activate one arm chosen uniformly from the set , regardless of the agent’s selection, and the reward is generated according to that arm’s reward distribution. We refer to this malfunction as -noise activation Based on the above data, answer the given subquestions.

Consider a faulty -armed bandit with -noise activation and distinct base success probabilities . Suppose the agent follows a greedy strategy that always chooses the arm with the highest empirical mean reward. How does the presence of activation noise affect the asymptotic fraction of time for which the optimal arm (with success probability ) is actually executed?

Consider an N-armed bandit problem in which each arm , for , produces a reward of 1 with probability and otherwise, where denotes the arm’s base success probability. Assume the bandit machine suffers from a hardware malfunction: when the agent attempts to pull arm , the intended action is not always executed. Instead, with probability a wiring fault causes the machine to randomly activate one arm chosen uniformly from the set , regardless of the agent’s selection, and the reward is generated according to that arm’s reward distribution. We refer to this malfunction as -noise activation Based on the above data, answer the given subquestions.

In a faulty -armed bandit with -noise activation, suppose you mistakenly model the environment as a standard bandit without activation noise. You estimate each arm’s success probability using sample averages of observed rewards. How does the -noise affect your estimates of the true base success probabilities ?

Consider a large-scale news recommendation platform that uses a contextual bandit approach to decide which headline to show to a user visiting the homepage. The context includes user features (e.g., location, device type, time of day, coarse interest profile) and article features (e.g., topic, recency, source reputation). Actions correspond to recommending one of the candidate headlines, and the reward is if the user clicks the recommended headline and otherwise. Let the context vector at time encode both user and article features for the current recommendation opportunity. There are discrete actions, each corresponding to selecting one of the candidate headlines for display. Based on the above data, answer the given subquestions.

Which of the following best explains why a contextual bandit approach is preferred over a standard multi-armed bandit in this news recommendation scenario?

Consider a large-scale news recommendation platform that uses a contextual bandit approach to decide which headline to show to a user visiting the homepage. The context includes user features (e.g., location, device type, time of day, coarse interest profile) and article features (e.g., topic, recency, source reputation). Actions correspond to recommending one of the candidate headlines, and the reward is if the user clicks the recommended headline and otherwise. Let the context vector at time encode both user and article features for the current recommendation opportunity. There are discrete actions, each corresponding to selecting one of the candidate headlines for display. Based on the above data, answer the given subquestions.

After many rounds of user interaction, what is the main optimality objective of the contextual bandit algorithm in this recommendation setting?

Consider a Markov Decision Process (MDP) with three states , , and , where is a terminal state and . Each non-terminal state has a single available action, and the transitions are deterministic with associated rewards: the agent moves from to and receives a reward of 1, and from to and receives a reward of 2. Let the discount factor be . Based on the above data, answer the given subquestions.

Consider a 2-state MDP with states and , discount factor . There are two actions in : and . Action gives immediate reward and moves deterministically to ; action gives immediate reward and keeps the agent in . In , there is a single action that yields an immediate reward and transitions back to . Assume initial value estimates are , Based on the above data, answer the given subquestions.

Consider an N-armed bandit problem in which each arm , for , produces a reward of 1 with probability and otherwise, where denotes the arm’s base success probability. Assume the bandit machine suffers from a hardware malfunction: when the agent attempts to pull arm , the intended action is not always executed. Instead, with probability a wiring fault causes the machine to randomly activate one arm chosen uniformly from the set , regardless of the agent’s selection, and the reward is generated according to that arm’s reward distribution. We refer to this malfunction as -noise activation Based on the above data, answer the given subquestions.

Consider a large-scale news recommendation platform that uses a contextual bandit approach to decide which headline to show to a user visiting the homepage. The context includes user features (e.g., location, device type, time of day, coarse interest profile) and article features (e.g., topic, recency, source reputation). Actions correspond to recommending one of the candidate headlines, and the reward is if the user clicks the recommended headline and otherwise. Let the context vector at time encode both user and article features for the current recommendation opportunity. There are discrete actions, each corresponding to selecting one of the candidate headlines for display. Based on the above data, answer the given subquestions.