A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detection

TL;DR

This paper introduces GW-VLM, a training-free vision-language model that achieves open-vocabulary object detection by combining multi-scale visual-language searching with contextual prompts, without any additional training.

Contribution

The paper proposes a novel training-free approach for OVOD using MS-VLS and CCP, enabling universal object understanding with pre-trained models without training.

Findings

Achieves superior OVOD performance on multiple datasets

Operates without any training step

Effective multi-scale visual-language alignment

Abstract

Open-Vocabulary Object Detection (OVOD) aims to develop the capability to detect anything. Although myriads of large-scale pre-training efforts have built versatile foundation models that exhibit impressive zero-shot capabilities to facilitate OVOD, the necessity of creating a universal understanding for any object cognition according to already pretrained foundation models is usually overlooked. Therefore, in this paper, a training-free Guess What Vision Language Model, called GW-VLM, is proposed to form a universal understanding paradigm based on our carefully designed Multi-Scale Visual Language Searching (MS-VLS) coupled with Contextual Concept Prompt (CCP) for OVOD. This approach can engage a pre-trained Vision Language Model (VLM) and a Large Language Model (LLM) in the game of "guess what". Wherein, MS-VLS leverages multi-scale visual-language soft-alignment for VLM to generate snippets from the results of class-agnostic object detection, while CCP can form the concept of flow referring to MS-VLS and then make LLM understand snippets for OVOD. Finally, the extensive experiments are carried out on natural and remote sensing datasets, including COCO val, Pascal VOC, DIOR, and NWPU-10, and the results indicate that our proposed GW-VLM can achieve superior OVOD performance compared to the-state-of-the-art methods without any training step.