Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networks

TL;DR

This paper investigates how overparameterization and the double descent phenomenon influence the generalization performance of transfer learning in deep neural networks, considering factors like dataset size, layer freezing, and task similarity.

Contribution

It provides a novel analysis of transfer learning through the lens of double descent and overparameterization, highlighting how these factors affect generalization and transfer success.

Findings

Double descent impacts transfer learning performance.

Larger source datasets slow target training.

Number of frozen layers influences under/overparameterization.

Abstract

We study the generalization behavior of transfer learning of deep neural networks (DNNs). We adopt the overparameterization perspective -- featuring interpolation of the training data (i.e., approximately zero train error) and the double descent phenomenon -- to explain the delicate effect of the transfer learning setting on generalization performance. We study how the generalization behavior of transfer learning is affected by the dataset size in the source and target tasks, the number of transferred layers that are kept frozen in the target DNN training, and the similarity between the source and target tasks. We show that the test error evolution during the target DNN training has a more significant double descent effect when the target training dataset is sufficiently large. In addition, a larger source training dataset can yield a slower target DNN training. Moreover, we demonstrate that the number of frozen layers can determine whether the transfer learning is effectively underparameterized or overparameterized and, in turn, this may induce a freezing-wise double descent phenomenon that determines the relative success or failure of learning. Also, we show that the double descent phenomenon may make a transfer from a less related source task better than a transfer from a more related source task. We establish our results using image classification experiments with the ResNet, DenseNet and the vision transformer (ViT) architectures.