Too Sharp, Too Sure: When Calibration Follows Curvature

TL;DR

This paper investigates the relationship between calibration, curvature, and margins in neural network training, proposing a margin-aware objective to improve calibration without losing accuracy.

Contribution

It reveals the coupling between calibration and curvature during training and introduces a new margin-aware training method to enhance calibration.

Findings

ECE closely tracks curvature-based sharpness during training

Both ECE and Gauss--Newton curvature are controlled by margin-dependent exponential tails

Margin-aware training improves out-of-sample calibration without sacrificing accuracy

Abstract

Modern neural networks can achieve high accuracy while remaining poorly calibrated, producing confidence estimates that do not match empirical correctness. Yet calibration is often treated as a post-hoc attribute. We take a different perspective: we study calibration as a training-time phenomenon on small vision tasks, and ask whether calibrated solutions can be obtained reliably by intervening on the training procedure. We identify a tight coupling between calibration, curvature, and margins during training of deep networks under multiple gradient-based methods. Empirically, Expected Calibration Error (ECE) closely tracks curvature-based sharpness throughout optimization. Mathematically, we show that both ECE and Gauss--Newton curvature are controlled, up to problem-specific constants, by the same margin-dependent exponential tail functional along the trajectory. Guided by this mechanism, we introduce a margin-aware training objective that explicitly targets robust-margin tails and local smoothness, yielding improved out-of-sample calibration across optimizers without sacrificing accuracy.