These Are Not All the Features You Are Looking For: A Fundamental Bottleneck in Supervised Pretraining

TL;DR

This paper identifies a fundamental bottleneck in supervised pretraining related to transfer bias caused by sparse, inconsistent features, and proposes an ensembling method to improve transfer accuracy without additional pretraining.

Contribution

It introduces a theoretical framework explaining transfer bias in pretrained models and proposes an inexpensive ensembling strategy to generate richer features for better transfer learning.

Findings

Ensembling improves transfer accuracy by 9% on ResNet.

Transfer bias is caused by inconsistent features learned during pretraining.

The phenomenon is widespread across modern deep learning architectures.

Abstract

Transfer learning is widely used to adapt large pretrained models to new tasks with only a small amount of new data. However, a challenge persists -- the features from the original task often do not fully cover what is needed for unseen data, especially when the relatedness of tasks is not clear. Since deep learning models tend to learn very sparse representations, they retain only the minimal features required for the initial training while discarding potentially ones for downstream transfer. A theoretical framework developed in this work demonstrates that such pretraining captures inconsistent aspects of the data distribution, therefore, inducing transfer bias. To address this limitation, we propose an inexpensive ensembling strategy that aggregates multiple models to generate richer feature representations. On ResNet, this approach yields a $9\%$ improvement in transfer accuracy without incurring extra pretraining cost. We also present empirical evidence from a range of deep learning studies, confirming that the phenomenon is pervasive across modern deep learning architectures. These results suggests that relying solely on large pretrained networks is not always the most effective way to improve model generalization. Instead, fostering richer, more diverse representations -- e.g. - through model ensembles -- can substantially enhance transfer learning performance.