On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines

TL;DR

This paper critically examines current calibration evaluation methods for object detectors, proposes a new evaluation framework, and demonstrates that simple post-hoc calibration methods outperform recent train-time approaches in calibration accuracy.

Contribution

The authors identify flaws in existing evaluation metrics, introduce a principled joint calibration-accuracy framework, and show that simple post-hoc calibrators are more effective than complex train-time methods.

Findings

Post-hoc calibrators outperform train-time calibration methods.

A new evaluation framework provides more accurate calibration assessment.

Isotonic Regression significantly reduces calibration error on COCO dataset.

Abstract

Reliable usage of object detectors require them to be calibrated -- a crucial problem that requires careful attention. Recent approaches towards this involve (1) designing new loss functions to obtain calibrated detectors by training them from scratch, and (2) post-hoc Temperature Scaling (TS) that learns to scale the likelihood of a trained detector to output calibrated predictions. These approaches are then evaluated based on a combination of Detection Expected Calibration Error (D-ECE) and Average Precision. In this work, via extensive analysis and insights, we highlight that these recent evaluation frameworks, evaluation metrics, and the use of TS have notable drawbacks leading to incorrect conclusions. As a step towards fixing these issues, we propose a principled evaluation framework to jointly measure calibration and accuracy of object detectors. We also tailor efficient and easy-to-use post-hoc calibration approaches such as Platt Scaling and Isotonic Regression specifically for object detection task. Contrary to the common notion, our experiments show that once designed and evaluated properly, post-hoc calibrators, which are extremely cheap to build and use, are much more powerful and effective than the recent train-time calibration methods. To illustrate, D-DETR with our post-hoc Isotonic Regression calibrator outperforms the recent train-time state-of-the-art calibration method Cal-DETR by more than 7 D-ECE on the COCO dataset. Additionally, we propose improved versions of the recently proposed Localization-aware ECE and show the efficacy of our method on these metrics as well. Code is available at: https://github.com/fiveai/detection_calibration.