Beyond Overconfidence: Foundation Models Redefine Calibration in Deep Neural Networks

TL;DR

This paper investigates the calibration properties of foundation models like ConvNeXt, EVA, and BEiT, revealing their tendencies towards underconfidence in-distribution and improved calibration under distribution shifts, with implications for deployment safety.

Contribution

It provides the first comprehensive analysis of foundation models' calibration behavior, challenging assumptions of continuous calibration improvements and evaluating post-hoc calibration methods under various conditions.

Findings

Foundation models are underconfident in in-distribution predictions.

Calibration improves under distribution shifts.

Post-hoc calibration methods are less reliable under severe shifts.

Abstract

Reliable uncertainty calibration is essential for safely deploying deep neural networks in high-stakes applications. Deep neural networks are known to exhibit systematic overconfidence, especially under distribution shifts. Although foundation models such as ConvNeXt, EVA and BEiT have demonstrated significant improvements in predictive performance, their calibration properties remain underexplored. This paper presents a comprehensive investigation into the calibration behavior of foundation models, revealing insights that challenge established paradigms. Our empirical analysis shows that these models tend to be underconfident in in-distribution predictions, resulting in higher calibration errors, while demonstrating improved calibration under distribution shifts. Furthermore, we demonstrate that foundation models are highly responsive to post-hoc calibration techniques in the in-distribution setting, enabling practitioners to effectively mitigate underconfidence bias. However, these methods become progressively less reliable under severe distribution shifts and can occasionally produce counterproductive results. Our findings highlight the complex, non-monotonic effects of architectural and training innovations on calibration, challenging established narratives of continuous improvement.