Bandits with Dynamic Arm-acquisition Costs

TL;DR

This paper studies a bandit problem where adding new arms incurs costs and affects the probability of finding optimal arms, proposing an adaptive algorithm with optimal long-term performance under certain conditions.

Contribution

It introduces a novel bandit model with dynamic arm acquisition costs and characterizes conditions for achieving sub-linear regret, along with a near-optimal adaptive algorithm.

Findings

Necessary condition for sub-linear regret identified

UCB-inspired algorithm shown to be long-run optimal

Conditions for achievability of optimal performance established

Abstract

We consider a bandit problem where at any time, the decision maker can add new arms to her consideration set. A new arm is queried at a cost from an "arm-reservoir" containing finitely many "arm-types," each characterized by a distinct mean reward. The cost of query reflects in a diminishing probability of the returned arm being optimal, unbeknown to the decision maker; this feature encapsulates defining characteristics of a broad class of operations-inspired online learning problems, e.g., those arising in markets with churn, or those involving allocations subject to costly resource acquisition. The decision maker's goal is to maximize her cumulative expected payoffs over a sequence of n pulls, oblivious to the statistical properties as well as types of the queried arms. We study two natural modes of endogeneity in the reservoir distribution, and characterize a necessary condition for achievability of sub-linear regret in the problem. We also discuss a UCB-inspired adaptive algorithm that is long-run-average optimal whenever said condition is satisfied, thereby establishing its tightness.