Open-Vocabulary Object Detection using Pseudo Caption Labels

TL;DR

This paper introduces Pseudo Caption Labeling (PCL), a simple method that uses image captioning models to generate dense, fine-grained labels for open-vocabulary object detection, improving knowledge transfer and generalization.

Contribution

The study proposes PCL, a novel pre-processing technique that leverages image captioning to produce detailed labels, enhancing open-vocabulary detection without altering existing models.

Findings

Achieved 34.5 AP on LVIS benchmark, comparable to state-of-the-art.

PCL improves detection of novel objects by providing richer, fine-grained labels.

Method is flexible and compatible with any captioning model.

Abstract

Recent open-vocabulary detection methods aim to detect novel objects by distilling knowledge from vision-language models (VLMs) trained on a vast amount of image-text pairs. To improve the effectiveness of these methods, researchers have utilized datasets with a large vocabulary that contains a large number of object classes, under the assumption that such data will enable models to extract comprehensive knowledge on the relationships between various objects and better generalize to unseen object classes. In this study, we argue that more fine-grained labels are necessary to extract richer knowledge about novel objects, including object attributes and relationships, in addition to their names. To address this challenge, we propose a simple and effective method named Pseudo Caption Labeling (PCL), which utilizes an image captioning model to generate captions that describe object instances from diverse perspectives. The resulting pseudo caption labels offer dense samples for knowledge distillation. On the LVIS benchmark, our best model trained on the de-duplicated VisualGenome dataset achieves an AP of 34.5 and an APr of 30.6, comparable to the state-of-the-art performance. PCL's simplicity and flexibility are other notable features, as it is a straightforward pre-processing technique that can be used with any image captioning model without imposing any restrictions on model architecture or training process.