Online Learning of Rested and Restless Bandits

TL;DR

This paper addresses online learning strategies for both rested and restless multiarmed bandits with multiple plays, focusing on maximizing long-term rewards in dynamic, uncertain environments such as spectrum access.

Contribution

It introduces new algorithms for online learning in multiarmed bandit settings with multiple plays and restless arms, applicable to spectrum access scenarios.

Findings

Developed algorithms with proven regret bounds

Demonstrated effectiveness in spectrum access simulations

Extended bandit models to restless environments

Abstract

In this paper we study the online learning problem involving rested and restless multiarmed bandits with multiple plays. The system consists of a single player/user and a set of K finite-state discrete-time Markov chains (arms) with unknown state spaces and statistics. At each time step the player can play M arms. The objective of the user is to decide for each step which M of the K arms to play over a sequence of trials so as to maximize its long term reward. The restless multiarmed bandit is particularly relevant to the application of opportunistic spectrum access (OSA), where a (secondary) user has access to a set of K channels, each of time-varying condition as a result of random fading and/or certain primary users' activities.