From COCO to COCO-FP: A Deep Dive into Background False Positives for COCO Detectors

TL;DR

This paper introduces COCO-FP, a new dataset derived from ImageNet-1K, to evaluate and analyze background false positives in object detectors, revealing significant room for improvement in reducing false alarms.

Contribution

The study presents COCO-FP, a novel dataset that specifically assesses background false positives in object detection, highlighting the need for improved models in real-world scenarios.

Findings

Significant false positives are present in standard detectors on COCO-FP.

AP50 for YOLOv9-E drops from 72.8 to 65.7 on COCO-FP.

Both closed-set and open-set scenarios show high false positive rates.

Abstract

Reducing false positives is essential for enhancing object detector performance, as reflected in the mean Average Precision (mAP) metric. Although object detectors have achieved notable improvements and high mAP scores on the COCO dataset, analysis reveals limited progress in addressing false positives caused by non-target visual clutter-background objects not included in the annotated categories. This issue is particularly critical in real-world applications, such as fire and smoke detection, where minimizing false alarms is crucial. In this study, we introduce COCO-FP, a new evaluation dataset derived from the ImageNet-1K dataset, designed to address this issue. By extending the original COCO validation dataset, COCO-FP specifically assesses object detectors' performance in mitigating background false positives. Our evaluation of both standard and advanced object detectors shows a significant number of false positives in both closed-set and open-set scenarios. For example, the AP50 metric for YOLOv9-E decreases from 72.8 to 65.7 when shifting from COCO to COCO-FP. The dataset is available at https://github.com/COCO-FP/COCO-FP.