MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection

TL;DR

MarvelOVD introduces a novel approach that combines object detection and vision-language models to improve open-vocabulary detection by refining pseudo-labels and addressing bias through online learning and stratified label assignment.

Contribution

The paper proposes MarvelOVD, a new paradigm that enhances pseudo-label quality and mitigates bias in open-vocabulary detection by integrating detector guidance with vision-language models.

Findings

Outperforms state-of-the-art methods on COCO and LVIS datasets.

Effectively reduces noisy pseudo-labels through Online Mining and Adaptive Reweighting.

Addresses the base-novel-conflict problem with stratified label assignments.

Abstract

Learning from pseudo-labels that generated with VLMs~(Vision Language Models) has been shown as a promising solution to assist open vocabulary detection (OVD) in recent studies. However, due to the domain gap between VLM and vision-detection tasks, pseudo-labels produced by the VLMs are prone to be noisy, while the training design of the detector further amplifies the bias. In this work, we investigate the root cause of VLMs' biased prediction under the OVD context. Our observations lead to a simple yet effective paradigm, coded MarvelOVD, that generates significantly better training targets and optimizes the learning procedure in an online manner by marrying the capability of the detector with the vision-language model. Our key insight is that the detector itself can act as a strong auxiliary guidance to accommodate VLM's inability of understanding both the ``background'' and the context of a proposal within the image. Based on it, we greatly purify the noisy pseudo-labels via Online Mining and propose Adaptive Reweighting to effectively suppress the biased training boxes that are not well aligned with the target object. In addition, we also identify a neglected ``base-novel-conflict'' problem and introduce stratified label assignments to prevent it. Extensive experiments on COCO and LVIS datasets demonstrate that our method outperforms the other state-of-the-arts by significant margins. Codes are available at https://github.com/wkfdb/MarvelOVD