Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

TL;DR

Fine-tuning pretrained models can harm out-of-distribution accuracy due to feature distortion, but a two-step linear probing then fine-tuning approach mitigates this issue, improving overall performance.

Contribution

This paper reveals the OOD performance degradation caused by fine-tuning and proposes a simple two-step method that combines the strengths of linear probing and fine-tuning.

Findings

Fine-tuning reduces OOD accuracy compared to linear probing.

LP-FT outperforms both fine-tuning and linear probing on multiple datasets.

Theoretical analysis explains feature distortion during fine-tuning.

Abstract

When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer -- the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A, ImageNet-Sketch), fine-tuning obtains on average 2% higher accuracy ID but 7% lower accuracy OOD than linear probing. We show theoretically that this tradeoff between ID and OOD accuracy arises even in a simple setting: fine-tuning overparameterized two-layer linear networks. We prove that the OOD error of fine-tuning is high when we initialize with a fixed or random head -- this is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features. Our analysis suggests that the easy two-step strategy of linear probing then full fine-tuning (LP-FT), sometimes used as a fine-tuning heuristic, combines the benefits of both fine-tuning and linear probing. Empirically, LP-FT outperforms both fine-tuning and linear probing on the above datasets (1% better ID, 10% better OOD than full fine-tuning).