Phase Transitions in Transfer Learning for High-Dimensional Perceptrons

TL;DR

This paper provides a theoretical analysis of transfer learning in high-dimensional perceptrons, revealing phase transitions between negative and positive transfer depending on task similarity.

Contribution

It introduces a simplified model that captures key phenomena of transfer learning, including the critical threshold for beneficial transfer.

Findings

Identifies a phase transition from negative to positive transfer

Defines a threshold of task similarity for effective transfer

Reproduces phenomena observed in practical transfer learning scenarios

Abstract

Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task. Central questions include deciding what information one should transfer and when transfer can be beneficial. The latter question is related to the so-called negative transfer phenomenon, where the transferred source information actually reduces the generalization performance of the target task. This happens when the two tasks are sufficiently dissimilar. In this paper, we present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks. Despite the simplicity of our model, it reproduces several key phenomena observed in practice. Specifically, our asymptotic analysis reveals a phase transition from negative transfer to positive transfer as the similarity of the two tasks moves past a well-defined threshold.