Visual Textualization for Image Prompted Object Detection

TL;DR

VisTex-OVLM introduces visual textualization, projecting visual exemplars into text space to improve object detection of rare categories in vision-language models, especially in few-shot scenarios.

Contribution

It presents a novel visual textualization technique that enhances OVLMs' ability to detect rare objects without altering their architecture.

Findings

Outperforms previous methods on open-set datasets.

Achieves state-of-the-art results on PASCAL VOC and MSCOCO.

Effective in few-shot object detection tasks.

Abstract

We propose VisTex-OVLM, a novel image prompted object detection method that introduces visual textualization -- a process that projects a few visual exemplars into the text feature space to enhance Object-level Vision-Language Models' (OVLMs) capability in detecting rare categories that are difficult to describe textually and nearly absent from their pre-training data, while preserving their pre-trained object-text alignment. Specifically, VisTex-OVLM leverages multi-scale textualizing blocks and a multi-stage fusion strategy to integrate visual information from visual exemplars, generating textualized visual tokens that effectively guide OVLMs alongside text prompts. Unlike previous methods, our method maintains the original architecture of OVLM, maintaining its generalization capabilities while enhancing performance in few-shot settings. VisTex-OVLM demonstrates superior performance across open-set datasets which have minimal overlap with OVLM's pre-training data and achieves state-of-the-art results on few-shot benchmarks PASCAL VOC and MSCOCO. The code will be released at https://github.com/WitGotFlg/VisTex-OVLM.