Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error

TL;DR

This paper introduces a differentiable surrogate for expected calibration error and a meta-learning framework to optimize neural network calibration directly, improving calibration quality with respect to model hyper-parameters.

Contribution

The paper presents a novel differentiable surrogate for expected calibration error and a meta-learning approach to optimize calibration on validation sets, advancing calibration techniques.

Findings

Achieves competitive calibration performance compared to existing methods.

Introduces a new differentiable surrogate for calibration error.

Provides a meta-learning framework for calibration optimization.

Abstract

Calibration of neural networks is a topical problem that is becoming more and more important as neural networks increasingly underpin real-world applications. The problem is especially noticeable when using modern neural networks, for which there is a significant difference between the confidence of the model and the probability of correct prediction. Various strategies have been proposed to improve calibration, yet accurate calibration remains challenging. We propose a novel framework with two contributions: introducing a new differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised, and a meta-learning framework that uses DECE to optimise for validation set calibration with respect to model hyper-parameters. The results show that we achieve competitive performance with existing calibration approaches. Our framework opens up a new avenue and toolset for tackling calibration, which we believe will inspire further work on this important challenge.