Scaling of Class-wise Training Losses for Post-hoc Calibration

TL;DR

This paper introduces a novel post-hoc calibration method that synchronizes class-wise training losses using multiple scaling factors, improving calibration performance especially on unbalanced datasets without sacrificing accuracy.

Contribution

The proposed framework effectively aligns class-wise training losses through multiple scaling factors, enhancing calibration while maintaining model performance and computational efficiency.

Findings

Improves calibration performance across various methods.

Performs well on unbalanced datasets.

Reduces additional computation in model calibration.

Abstract

The class-wise training losses often diverge as a result of the various levels of intra-class and inter-class appearance variation, and we find that the diverging class-wise training losses cause the uncalibrated prediction with its reliability. To resolve the issue, we propose a new calibration method to synchronize the class-wise training losses. We design a new training loss to alleviate the variance of class-wise training losses by using multiple class-wise scaling factors. Since our framework can compensate the training losses of overfitted classes with those of under-fitted classes, the integrated training loss is preserved, preventing the performance drop even after the model calibration. Furthermore, our method can be easily employed in the post-hoc calibration methods, allowing us to use the pre-trained model as an initial model and reduce the additional computation for model calibration. We validate the proposed framework by employing it in the various post-hoc calibration methods, which generally improves calibration performance while preserving accuracy, and discover through the investigation that our approach performs well with unbalanced datasets and untuned hyperparameters.