Feature Clipping for Uncertainty Calibration

TL;DR

This paper introduces feature clipping, a novel post-hoc method that improves neural network calibration by clipping feature values to reduce overconfidence, validated through extensive experiments and theoretical analysis.

Contribution

It presents the first feature-based post-hoc calibration technique, offering a new approach to enhance model calibration in deep neural networks.

Findings

Consistently improves calibration on CIFAR-10, CIFAR-100, and ImageNet.

Effective across CNNs and transformer architectures.

Outperforms existing post-hoc and train-time calibration methods.

Abstract

Deep neural networks (DNNs) have achieved significant success across various tasks, but ensuring reliable uncertainty estimates, known as model calibration, is crucial for their safe and effective deployment. Modern DNNs often suffer from overconfidence, leading to miscalibration. We propose a novel post-hoc calibration method called feature clipping (FC) to address this issue. FC involves clipping feature values to a specified threshold, effectively increasing entropy in high calibration error samples while maintaining the information in low calibration error samples. This process reduces the overconfidence in predictions, improving the overall calibration of the model. Our extensive experiments on datasets such as CIFAR-10, CIFAR-100, and ImageNet, and models including CNNs and transformers, demonstrate that FC consistently enhances calibration performance. Additionally, we provide a theoretical analysis that validates the effectiveness of our method. As the first calibration technique based on feature modification, feature clipping offers a novel approach to improving model calibration, showing significant improvements over both post-hoc and train-time calibration methods and pioneering a new avenue for feature-based model calibration.