Tabular and Deep Reinforcement Learning for Gittins Index

TL;DR

This paper introduces tabular and deep reinforcement learning algorithms to efficiently learn Gittins indices in multi-armed bandit problems with unknown Markovian dynamics, enabling better performance in large state spaces.

Contribution

It proposes novel RL algorithms (QGI and DGN) that are faster, require less memory, and show improved convergence for learning Gittins indices compared to existing methods.

Findings

Lower runtime and memory requirements

Better empirical convergence to Gittins index

Effective in large state space problems

Abstract

In the realm of multi-arm bandit problems, the Gittins index policy is known to be optimal in maximizing the expected total discounted reward obtained from pulling the Markovian arms. In most realistic scenarios however, the Markovian state transition probabilities are unknown and therefore the Gittins indices cannot be computed. One can then resort to reinforcement learning (RL) algorithms that explore the state space to learn these indices while exploiting to maximize the reward collected. In this work, we propose tabular (QGI) and Deep RL (DGN) algorithms for learning the Gittins index that are based on the retirement formulation for the multi-arm bandit problem. When compared with existing RL algorithms that learn the Gittins index, our algorithms have a lower run time, require less storage space (small Q-table size in QGI and smaller replay buffer in DGN), and illustrate better empirical convergence to the Gittins index. This makes our algorithm well suited for problems with large state spaces and is a viable alternative to existing methods. As a key application, we demonstrate the use of our algorithms in minimizing the mean flowtime in a job scheduling problem when jobs are available in batches and have an unknown service time distribution.