MaxEnt Loss: Constrained Maximum Entropy for Calibration under Out-of-Distribution Shift

TL;DR

This paper introduces MaxEnt Loss, a new calibration method based on maximum entropy principles that improves out-of-distribution calibration while maintaining accuracy, validated through theoretical analysis and empirical benchmarks.

Contribution

The paper proposes a novel MaxEnt Loss function that incorporates statistical constraints for better OOD calibration, outperforming existing methods.

Findings

Achieves state-of-the-art OOD calibration on benchmarks

Maintains model accuracy while improving calibration

Theoretically justified and empirically validated

Abstract

We present a new loss function that addresses the out-of-distribution (OOD) calibration problem. While many objective functions have been proposed to effectively calibrate models in-distribution, our findings show that they do not always fare well OOD. Based on the Principle of Maximum Entropy, we incorporate helpful statistical constraints observed during training, delivering better model calibration without sacrificing accuracy. We provide theoretical analysis and show empirically that our method works well in practice, achieving state-of-the-art calibration on both synthetic and real-world benchmarks.