An Underexplored Dilemma between Confidence and Calibration in Quantized Neural Networks

TL;DR

This paper investigates the relationship between confidence calibration and robustness to quantization in CNNs, revealing a trade-off where overconfidence can improve quantization robustness but worsen calibration.

Contribution

It uncovers the underexplored dilemma between confidence calibration and quantization robustness in CNNs, supported by empirical analysis on CIFAR-100 and ImageNet.

Findings

Overconfidence can enhance robustness to quantization.

Low confidence predictions are more likely to change after quantization.

High confidence predictions tend to be more accurate and stable.

Abstract

Modern convolutional neural networks (CNNs) are known to be overconfident in terms of their calibration on unseen input data. That is to say, they are more confident than they are accurate. This is undesirable if the probabilities predicted are to be used for downstream decision making. When considering accuracy, CNNs are also surprisingly robust to compression techniques, such as quantization, which aim to reduce computational and memory costs. We show that this robustness can be partially explained by the calibration behavior of modern CNNs, and may be improved with overconfidence. This is due to an intuitive result: low confidence predictions are more likely to change post-quantization, whilst being less accurate. High confidence predictions will be more accurate, but more difficult to change. Thus, a minimal drop in post-quantization accuracy is incurred. This presents a potential conflict in neural network design: worse calibration from overconfidence may lead to better robustness to quantization. We perform experiments applying post-training quantization to a variety of CNNs, on the CIFAR-100 and ImageNet datasets.