From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects

TL;DR

This paper introduces a novel framework for open world object detection that enables models to identify and learn new objects incrementally, overcoming limitations of traditional and open vocabulary detection methods, especially in autonomous driving scenarios.

Contribution

It proposes Open World Embedding Learning (OWEL) and Multi-Scale Contrastive Anchor Learning (MSCAL) to detect and learn unknown objects, improving open world detection performance.

Findings

Achieves state-of-the-art results on open world detection benchmarks.

Effectively detects far-out-of-distribution objects in autonomous driving.

Maintains open vocabulary detection capabilities while identifying unknown objects.

Abstract

Traditional object detection methods operate under the closed-set assumption, where models can only detect a fixed number of objects predefined in the training set. Recent works on open vocabulary object detection (OVD) enable the detection of objects defined by an in-principle unbounded vocabulary, which reduces the cost of training models for specific tasks. However, OVD heavily relies on accurate prompts provided by an ``oracle'', which limits their use in critical applications such as driving scene perception. OVD models tend to misclassify near-out-of-distribution (NOOD) objects that have similar features to known classes, and ignore far-out-of-distribution (FOOD) objects. To address these limitations, we propose a framework that enables OVD models to operate in open world settings, by identifying and incrementally learning previously unseen objects. To detect FOOD objects, we propose Open World Embedding Learning (OWEL) and introduce the concept of Pseudo Unknown Embedding which infers the location of unknown classes in a continuous semantic space based on the information of known classes. We also propose Multi-Scale Contrastive Anchor Learning (MSCAL), which enables the identification of misclassified unknown objects by promoting the intra-class consistency of object embeddings at different scales. The proposed method achieves state-of-the-art performance on standard open world object detection and autonomous driving benchmarks while maintaining its open vocabulary object detection capability.