The Ratio Index for Budgeted Learning, with Applications

TL;DR

This paper introduces the ratio index, a new index-based approach for budgeted learning and multi-armed bandits, providing constant factor approximations and efficient computation, improving upon existing algorithms and indices.

Contribution

The paper presents the ratio index, a novel index for budgeted learning that guarantees constant factor approximation and can be computed in strongly polynomial time.

Findings

The ratio index can be computed efficiently in strongly polynomial time.

It is a constant factor approximation to the Gittins index under certain conditions.

An index-based policy achieves an O(1)-approximation for finite horizon multi-armed bandits, independent of the horizon.

Abstract

In the budgeted learning problem, we are allowed to experiment on a set of alternatives (given a fixed experimentation budget) with the goal of picking a single alternative with the largest possible expected payoff. Approximation algorithms for this problem were developed by Guha and Munagala by rounding a linear program that couples the various alternatives together. In this paper we present an index for this problem, which we call the ratio index, which also guarantees a constant factor approximation. Index-based policies have the advantage that a single number (i.e. the index) can be computed for each alternative irrespective of all other alternatives, and the alternative with the highest index is experimented upon. This is analogous to the famous Gittins index for the discounted multi-armed bandit problem. The ratio index has several interesting structural properties. First, we show that it can be computed in strongly polynomial time. Second, we show that with the appropriate discount factor, the Gittins index and our ratio index are constant factor approximations of each other, and hence the Gittins index also gives a constant factor approximation to the budgeted learning problem. Finally, we show that the ratio index can be used to create an index-based policy that achieves an O(1)-approximation for the finite horizon version of the multi-armed bandit problem. Moreover, the policy does not require any knowledge of the horizon (whereas we compare its performance against an optimal strategy that is aware of the horizon). This yields the following surprising result: there is an index-based policy that achieves an O(1)-approximation for the multi-armed bandit problem, oblivious to the underlying discount factor.