Parametric $\rho$-Norm Scaling Calibration

TL;DR

This paper introduces a novel post-processing calibration method called $ ho$-Norm Scaling that improves uncertainty calibration in machine learning models by mitigating overconfidence and preserving instance-level information.

Contribution

The paper proposes a parametric calibration technique that expands the calibrator expression and incorporates distribution regularization to enhance uncertainty calibration.

Findings

Significant improvement in uncertainty calibration performance.

Effective mitigation of overconfidence due to amplitude amplification.

Preservation of instance-level uncertainty distribution after calibration.

Abstract

Output uncertainty indicates whether the probabilistic properties reflect objective characteristics of the model output. Unlike most loss functions and metrics in machine learning, uncertainty pertains to individual samples, but validating it on individual samples is unfeasible. When validated collectively, it cannot fully represent individual sample properties, posing a challenge in calibrating model confidence in a limited data set. Hence, it is crucial to consider confidence calibration characteristics. To counter the adverse effects of the gradual amplification of the classifier output amplitude in supervised learning, we introduce a post-processing parametric calibration method, $\rho$-Norm Scaling, which expands the calibrator expression and mitigates overconfidence due to excessive amplitude while preserving accuracy. Moreover, bin-level objective-based calibrator optimization often results in the loss of significant instance-level information. Therefore, we include probability distribution regularization, which incorporates specific priori information that the instance-level uncertainty distribution after calibration should resemble the distribution before calibration. Experimental results demonstrate the substantial enhancement in the post-processing calibrator for uncertainty calibration with our proposed method.