Open World Object Detection in the Era of Foundation Models

TL;DR

This paper explores the use of foundation models for open world object detection, introduces a new challenging benchmark with real-world datasets, and proposes FOMO, a method that significantly improves unknown object detection performance.

Contribution

It presents a new benchmark for evaluating foundation model-based open world object detection and introduces FOMO, a novel method leveraging class attribute sharing for better unknown object detection.

Findings

FOMO achieves ~3x higher unknown object mAP than baselines.

Existing benchmarks are insufficient for evaluating foundation model integration.

A new benchmark with diverse real-world datasets was curated.

Abstract

Object detection is integral to a bevy of real-world applications, from robotics to medical image analysis. To be used reliably in such applications, models must be capable of handling unexpected - or novel - objects. The open world object detection (OWD) paradigm addresses this challenge by enabling models to detect unknown objects and learn discovered ones incrementally. However, OWD method development is hindered due to the stringent benchmark and task definitions. These definitions effectively prohibit foundation models. Here, we aim to relax these definitions and investigate the utilization of pre-trained foundation models in OWD. First, we show that existing benchmarks are insufficient in evaluating methods that utilize foundation models, as even naive integration methods nearly saturate these benchmarks. This result motivated us to curate a new and challenging benchmark for these models. Therefore, we introduce a new benchmark that includes five real-world application-driven datasets, including challenging domains such as aerial and surgical images, and establish baselines. We exploit the inherent connection between classes in application-driven datasets and introduce a novel method, Foundation Object detection Model for the Open world, or FOMO, which identifies unknown objects based on their shared attributes with the base known objects. FOMO has ~3x unknown object mAP compared to baselines on our benchmark. However, our results indicate a significant place for improvement - suggesting a great research opportunity in further scaling object detection methods to real-world domains. Our code and benchmark are available at https://orrzohar.github.io/projects/fomo/.