Calibrating Deep Neural Networks using Explicit Regularisation and Dynamic Data Pruning

TL;DR

This paper introduces a novel regularization method combined with dynamic data pruning during training to improve neural network calibration, especially for high-confidence predictions, enhancing deployment safety and efficiency.

Contribution

The paper proposes a new regularization technique and a dynamic pruning strategy that together achieve state-of-the-art calibration and increase high-confidence predictions in neural networks.

Findings

State-of-the-art calibration on image benchmarks

Reduces training time significantly

Increases high-confidence predictions at test time

Abstract

Deep neural networks (DNN) are prone to miscalibrated predictions, often exhibiting a mismatch between the predicted output and the associated confidence scores. Contemporary model calibration techniques mitigate the problem of overconfident predictions by pushing down the confidence of the winning class while increasing the confidence of the remaining classes across all test samples. However, from a deployment perspective, an ideal model is desired to (i) generate well-calibrated predictions for high-confidence samples with predicted probability say >0.95, and (ii) generate a higher proportion of legitimate high-confidence samples. To this end, we propose a novel regularization technique that can be used with classification losses, leading to state-of-the-art calibrated predictions at test time; From a deployment standpoint in safety-critical applications, only high-confidence samples from a well-calibrated model are of interest, as the remaining samples have to undergo manual inspection. Predictive confidence reduction of these potentially ``high-confidence samples'' is a downside of existing calibration approaches. We mitigate this by proposing a dynamic train-time data pruning strategy that prunes low-confidence samples every few epochs, providing an increase in "confident yet calibrated samples". We demonstrate state-of-the-art calibration performance across image classification benchmarks, reducing training time without much compromise in accuracy. We provide insights into why our dynamic pruning strategy that prunes low-confidence training samples leads to an increase in high-confidence samples at test time.