Adaptive Temperature Scaling for Robust Calibration of Deep Neural Networks

TL;DR

This paper introduces Adaptive Temperature Scaling, a post-hoc calibration method for neural networks that balances expressiveness and robustness, especially in data-scarce scenarios, and provides interpretability of the calibration process.

Contribution

It proposes Entropy-based Temperature Scaling, a simple yet effective method that achieves state-of-the-art calibration performance and robustness against limited data, with improved interpretability.

Findings

Entropy-based Temperature Scaling outperforms existing methods.

The method is robust when data is scarce.

It offers better interpretability of calibration.

Abstract

In this paper, we study the post-hoc calibration of modern neural networks, a problem that has drawn a lot of attention in recent years. Many calibration methods of varying complexity have been proposed for the task, but there is no consensus about how expressive these should be. We focus on the task of confidence scaling, specifically on post-hoc methods that generalize Temperature Scaling, we call these the Adaptive Temperature Scaling family. We analyse expressive functions that improve calibration and propose interpretable methods. We show that when there is plenty of data complex models like neural networks yield better performance, but are prone to fail when the amount of data is limited, a common situation in certain post-hoc calibration applications like medical diagnosis. We study the functions that expressive methods learn under ideal conditions and design simpler methods but with a strong inductive bias towards these well-performing functions. Concretely, we propose Entropy-based Temperature Scaling, a simple method that scales the confidence of a prediction according to its entropy. Results show that our method obtains state-of-the-art performance when compared to others and, unlike complex models, it is robust against data scarcity. Moreover, our proposed model enables a deeper interpretation of the calibration process.