Enhancing Open-Vocabulary Object Detection through Multi-Level Fine-Grained Visual-Language Alignment

TL;DR

This paper introduces VLDet, a novel framework that enhances open-vocabulary object detection by improving multi-level visual-language alignment, leading to state-of-the-art results on COCO and LVIS datasets.

Contribution

VLDet revamps feature pyramids for fine-grained visual-language alignment and introduces SigRPN with contrastive loss, advancing open-vocabulary object detection capabilities.

Findings

Achieves 58.7 AP on COCO novel classes

Surpasses state-of-the-art on LVIS with 24.8 AP

Demonstrates superior zero-shot detection performance

Abstract

Traditional object detection systems are typically constrained to predefined categories, limiting their applicability in dynamic environments. In contrast, open-vocabulary object detection (OVD) enables the identification of objects from novel classes not present in the training set. Recent advances in visual-language modeling have led to significant progress of OVD. However, prior works face challenges in either adapting the single-scale image backbone from CLIP to the detection framework or ensuring robust visual-language alignment. We propose Visual-Language Detection (VLDet), a novel framework that revamps feature pyramid for fine-grained visual-language alignment, leading to improved OVD performance. With the VL-PUB module, VLDet effectively exploits the visual-language knowledge from CLIP and adapts the backbone for object detection through feature pyramid. In addition, we introduce the SigRPN block, which incorporates a sigmoid-based anchor-text contrastive alignment loss to improve detection of novel categories. Through extensive experiments, our approach achieves 58.7 AP for novel classes on COCO2017 and 24.8 AP on LVIS, surpassing all state-of-the-art methods and achieving significant improvements of 27.6% and 6.9%, respectively. Furthermore, VLDet also demonstrates superior zero-shot performance on closed-set object detection.