What is being transferred in transfer learning?

TL;DR

This paper investigates the mechanisms behind transfer learning in deep neural networks, distinguishing between feature reuse and learning data statistics, and analyzing the effects of pre-training on model behavior.

Contribution

It introduces new analytical tools to understand what aspects of models are transferred and how pre-trained weights influence model stability and similarity.

Findings

Transfer learning benefits partly from learning data statistics.

Pre-trained models tend to stay in the same loss landscape basin.

Models initialized with pre-trained weights are similar in feature and parameter space.

Abstract

One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learning in various deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analyses to address these fundamental questions. Through a series of analyses on transferring to block-shuffled images, we separate the effect of feature reuse from learning low-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.