A PAC-Bayesian bound for Lifelong Learning

TL;DR

This paper introduces a PAC-Bayesian generalization bound for lifelong learning, providing a theoretical framework that unifies various transfer learning paradigms and guides the development of new algorithms.

Contribution

It presents the first PAC-Bayesian bound tailored for lifelong learning, unifying transfer learning paradigms and deriving new algorithms with competitive results.

Findings

The bound offers a unified theoretical view of transfer learning methods.

Derived algorithms perform comparably to existing approaches.

Theoretical insights guide the design of lifelong learning algorithms.

Abstract

Transfer learning has received a lot of attention in the machine learning community over the last years, and several effective algorithms have been developed. However, relatively little is known about their theoretical properties, especially in the setting of lifelong learning, where the goal is to transfer information to tasks for which no data have been observed so far. In this work we study lifelong learning from a theoretical perspective. Our main result is a PAC-Bayesian generalization bound that offers a unified view on existing paradigms for transfer learning, such as the transfer of parameters or the transfer of low-dimensional representations. We also use the bound to derive two principled lifelong learning algorithms, and we show that these yield results comparable with existing methods.