A General Framework of Multi-Armed Bandit Processes by Arm Switch Restrictions

TL;DR

This paper introduces a unified framework for multi-armed bandit processes with switch restrictions, extending classical models and simplifying the proof of Gittins index policy optimality.

Contribution

It develops a general theory for MAB processes with switch restrictions, unifying various existing models and introducing new proof techniques for Gittins index optimality.

Findings

Gittins index process constructed under switch restrictions

Optimality of Gittins index rule established in the new framework

Framework encompasses classical and new MAB models

Abstract

This paper proposes a general framework of multi-armed bandit (MAB) processes by introducing a type of restrictions on the switches among arms evolving in continuous time. The Gittins index process is constructed for any single arm subject to the restrictions on switches and then the optimality of the corresponding Gittins index rule is established. The Gittins indices defined in this paper are consistent with the ones for MAB processes in continuous time, integer time, semi-Markovian setting as well as general discrete time setting, so that the new theory covers the classical models as special cases and also applies to many other situations that have not yet been touched in the literature. While the proof of the optimality of Gittins index policies benefits from ideas in the existing theory of MAB processes in continuous time, new techniques are introduced which drastically simplify the proof.