On the Potential of Open-Vocabulary Models for Object Detection in Unusual Street Scenes

TL;DR

This paper evaluates the effectiveness of open-vocabulary object detection models in identifying unusual and out-of-distribution objects in street scenes, highlighting their potential and current limitations for real-world applications.

Contribution

The study benchmarks four state-of-the-art open-vocabulary object detectors across three datasets, revealing their strengths and shortcomings in challenging street scene scenarios.

Findings

Grounding DINO achieves top AP of 48.3% on RoadObstacle21.

YOLO-World achieves 21.2% AP on RoadAnomaly21.

Open-vocabulary models show promise but require improvements for reliable deployment.

Abstract

Out-of-distribution (OOD) object detection is a critical task focused on detecting objects that originate from a data distribution different from that of the training data. In this study, we investigate to what extent state-of-the-art open-vocabulary object detectors can detect unusual objects in street scenes, which are considered as OOD or rare scenarios with respect to common street scene datasets. Specifically, we evaluate their performance on the OoDIS Benchmark, which extends RoadAnomaly21 and RoadObstacle21 from SegmentMeIfYouCan, as well as LostAndFound, which was recently extended to object level annotations. The objective of our study is to uncover short-comings of contemporary object detectors in challenging real-world, and particularly in open-world scenarios. Our experiments reveal that open vocabulary models are promising for OOD object detection scenarios, however far from perfect. Substantial improvements are required before they can be reliably deployed in real-world applications. We benchmark four state-of-the-art open-vocabulary object detection models on three different datasets. Noteworthily, Grounding DINO achieves the best results on RoadObstacle21 and LostAndFound in our study with an AP of 48.3% and 25.4% respectively. YOLO-World excels on RoadAnomaly21 with an AP of 21.2%.