Robust Experimentation in the Continuous Time Bandit Problem

TL;DR

This paper analyzes how a decision maker optimally experiments in a two-armed bandit problem with ambiguity, using continuous-time differential game methods to derive strategies and the impact of additional information on exploration behavior.

Contribution

It introduces a novel continuous-time framework for bandit experimentation with ambiguity and derives explicit thresholds for exploration based on ambiguity aversion.

Findings

Optimal exploration threshold increases with ambiguity aversion.

Provision of unambiguous information raises the exploration threshold.

The decision maker adopts a cut-off strategy based on belief thresholds.

Abstract

We study the experimentation dynamics of a decision maker (DM) in a two-armed bandit setup (Bolton and Harris (1999)), where the agent holds ambiguous beliefs regarding the distribution of the return process of one arm and is certain about the other one. The DM entertains Multiplier preferences a la Hansen and Sargent (2001), thus we frame the decision making environment as a two-player differential game against nature in continuous time. We characterize the DM value function and her optimal experimentation strategy that turns out to follow a cut-off rule with respect to her belief process. The belief threshold for exploring the ambiguous arm is found in closed form and is shown to be increasing with respect to the ambiguity aversion index. We then study the effect of provision of an unambiguous information source about the ambiguous arm. Interestingly, we show that the exploration threshold rises unambiguously as a result of this new information source, thereby leading to more conservatism. This analysis also sheds light on the efficient time to reach for an expert opinion.