An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration

TL;DR

This study empirically evaluates how pre-trained model size, dataset, and training strategies influence out-of-distribution generalization and calibration, revealing the critical importance of model selection for robust performance.

Contribution

It systematically analyzes the impact of pre-trained model characteristics on OOD performance and calibration, highlighting the significance of model selection over algorithm improvements.

Findings

Larger models and datasets improve OOD accuracy and calibration.

Optimal pre-trained model choices significantly outperform algorithm-focused methods.

Modern deep networks can have better calibration than shallow models, contrary to prior beliefs.

Abstract

In the field of computer vision, fine-tuning pre-trained models has become a prevalent strategy for out-of-distribution (OOD) generalization tasks. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and confidence calibration on downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pre-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary to some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for out-of-distribution generalization and calibration.