PAC-Bayesian Lifelong Learning For Multi-Armed Bandits

TL;DR

This paper introduces a PAC-Bayesian framework for lifelong learning in multi-armed bandit problems, deriving bounds and algorithms that improve transfer learning across sequential tasks.

Contribution

It provides the first PAC-Bayesian analysis tailored for lifelong learning in multi-armed bandits, with new bounds and algorithms leveraging these bounds for better transfer learning.

Findings

Proposed algorithms outperform baseline methods in experiments.

Derived lower bounds on expected rewards in lifelong bandit settings.

Validated effectiveness of bounds-based algorithms across multiple tasks.

Abstract

We present a PAC-Bayesian analysis of lifelong learning. In the lifelong learning problem, a sequence of learning tasks is observed one-at-a-time, and the goal is to transfer information acquired from previous tasks to new learning tasks. We consider the case when each learning task is a multi-armed bandit problem. We derive lower bounds on the expected average reward that would be obtained if a given multi-armed bandit algorithm was run in a new task with a particular prior and for a set number of steps. We propose lifelong learning algorithms that use our new bounds as learning objectives. Our proposed algorithms are evaluated in several lifelong multi-armed bandit problems and are found to perform better than a baseline method that does not use generalisation bounds.