Asymptotic Allocation Rules for a Class of Dynamic Multi-armed Bandit Problems

TL;DR

This paper introduces a unified framework for a class of dynamic multi-armed bandit problems with time-varying rewards modeled as noisy linear stochastic systems, enabling analysis of complex, practical scenarios.

Contribution

It develops asymptotic allocation rules ensuring logarithmic regret bounds for this class, combining UCB algorithms with reward estimators, and demonstrates practical applicability.

Findings

Logarithmic bounds on expected cumulative regret.

Versatility of the approach for various practical problems.

Effective combination of UCB algorithms with reward estimators.

Abstract

This paper presents a class of Dynamic Multi-Armed Bandit problems where the reward can be modeled as the noisy output of a time varying linear stochastic dynamic system that satisfies some boundedness constraints. The class allows many seemingly different problems with time varying option characteristics to be considered in a single framework. It also opens up the possibility of considering many new problems of practical importance. For instance it affords the simultaneous consideration of temporal option unavailabilities and the depen- dencies between options with time varying option characteristics in a seamless manner. We show that, for this class of problems, the combination of any Upper Confidence Bound type algorithm with any efficient reward estimator for the expected reward ensures the logarithmic bounding of the expected cumulative regret. We demonstrate the versatility of the approach by the explicit consideration of a new example of practical interest.