Automated Model Evaluation for Object Detection via Prediction Consistency and Reliability

TL;DR

This paper introduces AutoEval, an automated framework for evaluating object detection models by measuring prediction consistency and reliability without ground-truth labels, using a novel meta-dataset with image corruptions.

Contribution

We propose PCR, a new method leveraging multiple candidate boxes to estimate detection performance without labels, and create a diverse meta-dataset for realistic evaluation.

Findings

PCR outperforms existing AutoEval methods in accuracy

Meta-dataset covers a wider range of detection performance

Evaluation is scalable and does not require manual annotation

Abstract

Recent advances in computer vision have made training object detectors more efficient and effective; however, assessing their performance in real-world applications still relies on costly manual annotation. To address this limitation, we develop an automated model evaluation (AutoEval) framework for object detection. We propose Prediction Consistency and Reliability (PCR), which leverages the multiple candidate bounding boxes that conventional detectors generate before non-maximum suppression (NMS). PCR estimates detection performance without ground-truth labels by jointly measuring 1) the spatial consistency between boxes before and after NMS, and 2) the reliability of the retained boxes via the confidence scores of overlapping boxes. For a more realistic and scalable evaluation, we construct a meta-dataset by applying image corruptions of varying severity. Experimental results demonstrate that PCR yields more accurate performance estimates than existing AutoEval methods, and the proposed meta-dataset covers a wider range of detection performance. The code is available at https://github.com/YonseiML/autoeval-det.