Question 15
Consider an N-armed bandit problem in which each arm , for , produces a reward of 1 with probability and otherwise, where denotes the arm’s base success probability. Assume the bandit machine suffers from a hardware malfunction: when the agent attempts to pull arm , the intended action is not always executed. Instead, with probability a wiring fault causes the machine to randomly activate one arm chosen uniformly from the set , regardless of the agent’s selection, and the reward is generated according to that arm’s reward distribution. We refer to this malfunction as -noise activation Based on the above data, answer the given subquestions.
Consider a faulty -armed bandit with -noise activation and distinct base success probabilities . Suppose the agent follows a greedy strategy that always chooses the arm with the highest empirical mean reward. How does the presence of activation noise affect the asymptotic fraction of time for which the optimal arm (with success probability ) is actually executed?
Noise has no effect; the optimal arm is activated almost surely in the long run.
Noise reduces the frequency with which the optimal arm is selected, but once it is selected, it is always activated.
Noise eventually causes the greedy policy to oscillate indefinitely between arms, preventing convergence.
Even when the greedy policy converges to always selecting the optimal arm, activation noise prevents the optimal arm from being activated more than a fraction of the time.