From Black-box to White-box: Examining Confidence Calibration under different Conditions

TL;DR

This paper investigates how post-processing techniques like non-maximum suppression impact the confidence calibration of object detection models, revealing that such methods can degrade initial calibration and cause overconfidence.

Contribution

It introduces an analysis of calibration in object detection considering both black-box and white-box approaches, highlighting the effects of post-processing on confidence calibration.

Findings

Post-processing significantly affects confidence calibration.

Non-maximum suppression can lead to overconfident predictions.

Calibration is influenced by image location and box scale.

Abstract

Confidence calibration is a major concern when applying artificial neural networks in safety-critical applications. Since most research in this area has focused on classification in the past, confidence calibration in the scope of object detection has gained more attention only recently. Based on previous work, we study the miscalibration of object detection models with respect to image location and box scale. Our main contribution is to additionally consider the impact of box selection methods like non-maximum suppression to calibration. We investigate the default intrinsic calibration of object detection models and how it is affected by these post-processing techniques. For this purpose, we distinguish between black-box calibration with non-maximum suppression and white-box calibration with raw network outputs. Our experiments reveal that post-processing highly affects confidence calibration. We show that non-maximum suppression has the potential to degrade initially well-calibrated predictions, leading to overconfident and thus miscalibrated models.