Test-time Vocabulary Adaptation for Language-driven Object Detection

TL;DR

This paper introduces VocAda, a plug-and-play method that refines user-defined vocabularies at test time for open-vocabulary object detection, improving accuracy without additional training.

Contribution

The paper presents VocAda, a novel inference-time vocabulary adaptation method that automatically filters relevant classes, enhancing detection performance in open-vocabulary settings.

Findings

Consistently improves detection accuracy on COCO and Objects365 datasets.

Operates without additional training, making it versatile and easy to integrate.

Effective across multiple state-of-the-art detectors.

Abstract

Open-vocabulary object detection models allow users to freely specify a class vocabulary in natural language at test time, guiding the detection of desired objects. However, vocabularies can be overly broad or even mis-specified, hampering the overall performance of the detector. In this work, we propose a plug-and-play Vocabulary Adapter (VocAda) to refine the user-defined vocabulary, automatically tailoring it to categories that are relevant for a given image. VocAda does not require any training, it operates at inference time in three steps: i) it uses an image captionner to describe visible objects, ii) it parses nouns from those captions, and iii) it selects relevant classes from the user-defined vocabulary, discarding irrelevant ones. Experiments on COCO and Objects365 with three state-of-the-art detectors show that VocAda consistently improves performance, proving its versatility. The code is open source.