Understanding the Role of Invariance in Transfer Learning

TL;DR

This paper systematically investigates how invariance in representations affects transfer learning, revealing that the right invariances are crucial for high transfer performance and exploring their transferability across tasks.

Contribution

It introduces synthetic datasets to precisely analyze the role of invariance in transfer learning and identifies when invariance helps or hinders transfer performance.

Findings

Invariance to key transformations is often more important than model size or data quantity.

Invariance can sometimes negatively impact transferability depending on the task.

Transferability of invariance varies and depends on the relationship between source and target tasks.

Abstract

Transfer learning is a powerful technique for knowledge-sharing between different tasks. Recent work has found that the representations of models with certain invariances, such as to adversarial input perturbations, achieve higher performance on downstream tasks. These findings suggest that invariance may be an important property in the context of transfer learning. However, the relationship of invariance with transfer performance is not fully understood yet and a number of questions remain. For instance, how important is invariance compared to other factors of the pretraining task? How transferable is learned invariance? In this work, we systematically investigate the importance of representational invariance for transfer learning, as well as how it interacts with other parameters during pretraining. To do so, we introduce a family of synthetic datasets that allow us to precisely control factors of variation both in training and test data. Using these datasets, we a) show that for learning representations with high transfer performance, invariance to the right transformations is as, or often more, important than most other factors such as the number of training samples, the model architecture and the identity of the pretraining classes, b) show conditions under which invariance can harm the ability to transfer representations and c) explore how transferable invariance is between tasks. The code is available at \url{https://github.com/tillspeicher/representation-invariance-transfer}.