How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection

TL;DR

This paper introduces OVDEval, a comprehensive benchmark with 9 sub-tasks and a new evaluation metric NMS-AP to better assess the generalization and understanding capabilities of open-vocabulary object detection models.

Contribution

The paper presents a new benchmark dataset with fine-grained tasks and a novel evaluation metric addressing limitations of existing methods in open-vocabulary detection.

Findings

Existing top OVD models struggle on new fine-grained tasks.

NMS-AP provides more accurate evaluation than traditional AP.

Benchmark reveals weaknesses in current OVD models.

Abstract

Object detection (OD) in computer vision has made significant progress in recent years, transitioning from closed-set labels to open-vocabulary detection (OVD) based on large-scale vision-language pre-training (VLP). However, current evaluation methods and datasets are limited to testing generalization over object types and referral expressions, which do not provide a systematic, fine-grained, and accurate benchmark of OVD models' abilities. In this paper, we propose a new benchmark named OVDEval, which includes 9 sub-tasks and introduces evaluations on commonsense knowledge, attribute understanding, position understanding, object relation comprehension, and more. The dataset is meticulously created to provide hard negatives that challenge models' true understanding of visual and linguistic input. Additionally, we identify a problem with the popular Average Precision (AP) metric when benchmarking models on these fine-grained label datasets and propose a new metric called Non-Maximum Suppression Average Precision (NMS-AP) to address this issue. Extensive experimental results show that existing top OVD models all fail on the new tasks except for simple object types, demonstrating the value of the proposed dataset in pinpointing the weakness of current OVD models and guiding future research. Furthermore, the proposed NMS-AP metric is verified by experiments to provide a much more truthful evaluation of OVD models, whereas traditional AP metrics yield deceptive results. Data is available at \url{https://github.com/om-ai-lab/OVDEval}