Two-Armed Restless Bandits with Imperfect Information: Stochastic Control and Indexability

TL;DR

This paper introduces a new class of restless bandit models with increasing and decreasing payoffs for different arms, showing that optimal strategies are index-based stopping rules and establishing indexability using stochastic control insights.

Contribution

It develops a novel restless bandit model with specific payoff dynamics and proves indexability, extending Gittins' index theory to this new setting.

Findings

Optimal strategies are characterized as index-based stopping rules.

The model demonstrates indexability of a new class of restless bandits.

The approach connects stochastic control with bandit indexability.

Abstract

We present a two-armed bandit model of decision making under uncertainty where the expected return to investing in the "risky arm" increases when choosing that arm and decreases when choosing the "safe" arm. These dynamics are natural in applications such as human capital development, job search, and occupational choice. Using new insights from stochastic control, along with a monotonicity condition on the payoff dynamics, we show that optimal strategies in our model are stopping rules that can be characterized by an index which formally coincides with Gittins' index. Our result implies the indexability of a new class of restless bandit models.