Limits of Transfer Learning

TL;DR

This paper develops theoretical bounds and insights into transfer learning, emphasizing the importance of selecting relevant transferred information and understanding the limits of improvement based on probabilistic changes.

Contribution

It provides novel theoretical results that clarify when transfer learning is effective and how the degree of change impacts potential gains, advancing the understanding of transfer learning's limitations.

Findings

Dependence between transferred info and target problems is crucial.

Upper bounds on transfer learning improvements are established.

The results apply broadly within the algorithmic search framework.

Abstract

Transfer learning involves taking information and insight from one problem domain and applying it to a new problem domain. Although widely used in practice, theory for transfer learning remains less well-developed. To address this, we prove several novel results related to transfer learning, showing the need to carefully select which sets of information to transfer and the need for dependence between transferred information and target problems. Furthermore, we prove how the degree of probabilistic change in an algorithm using transfer learning places an upper bound on the amount of improvement possible. These results build on the algorithmic search framework for machine learning, allowing the results to apply to a wide range of learning problems using transfer.