Open-vocabulary vs. Closed-set: Best Practice for Few-shot Object Detection Considering Text Describability

TL;DR

This paper evaluates the effectiveness of open-vocabulary versus closed-set object detection in few-shot scenarios, introducing a measure of text-describability to guide dataset categorization and method selection.

Contribution

It proposes a novel way to quantify dataset text-describability and empirically compares OVD and COD methods across different dataset categories.

Findings

OVD and COD perform similarly on low text-describability classes.

Increasing training data volume with OVD can be counterproductive for low-describability classes.

The proposed measure helps guide practitioners in choosing appropriate detection methods.

Abstract

Open-vocabulary object detection (OVD), detecting specific classes of objects using only their linguistic descriptions (e.g., class names) without any image samples, has garnered significant attention. However, in real-world applications, the target class concepts is often hard to describe in text and the only way to specify target objects is to provide their image examples, yet it is often challenging to obtain a good number of samples. Thus, there is a high demand from practitioners for few-shot object detection (FSOD). A natural question arises: Can the benefits of OVD extend to FSOD for object classes that are difficult to describe in text? Compared to traditional methods that learn only predefined classes (referred to in this paper as closed-set object detection, COD), can the extra cost of OVD be justified? To answer these questions, we propose a method to quantify the ``text-describability'' of object detection datasets using the zero-shot image classification accuracy with CLIP. This allows us to categorize various OD datasets with different text-describability and emprically evaluate the FSOD performance of OVD and COD methods within each category. Our findings reveal that: i) there is little difference between OVD and COD for object classes with low text-describability under equal conditions in OD pretraining; and ii) although OVD can learn from more diverse data than OD-specific data, thereby increasing the volume of training data, it can be counterproductive for classes with low-text-describability. These findings provide practitioners with valuable guidance amidst the recent advancements of OVD methods.