DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection

TL;DR

DetCLIP introduces a knowledge-enriched, dictionary-based pre-training approach for open-world object detection, significantly improving zero-shot detection performance by leveraging concept descriptions and relationships.

Contribution

It proposes a novel parallel concept formulation and a comprehensive concept dictionary to enhance open-world detection and zero-shot learning capabilities.

Findings

DetCLIP-T outperforms GLIP-T by 9.9% mAP on LVIS.

Achieves 13.5% improvement on rare categories.

Demonstrates strong zero-shot detection performance.

Abstract

Open-world object detection, as a more general and challenging goal, aims to recognize and localize objects described by arbitrary category names. The recent work GLIP formulates this problem as a grounding problem by concatenating all category names of detection datasets into sentences, which leads to inefficient interaction between category names. This paper presents DetCLIP, a paralleled visual-concept pre-training method for open-world detection by resorting to knowledge enrichment from a designed concept dictionary. To achieve better learning efficiency, we propose a novel paralleled concept formulation that extracts concepts separately to better utilize heterogeneous datasets (i.e., detection, grounding, and image-text pairs) for training. We further design a concept dictionary~(with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept. By enriching the concepts with their descriptions, we explicitly build the relationships among various concepts to facilitate the open-domain learning. The proposed concept dictionary is further used to provide sufficient negative concepts for the construction of the word-region alignment loss\, and to complete labels for objects with missing descriptions in captions of image-text pair data. The proposed framework demonstrates strong zero-shot detection performances, e.g., on the LVIS dataset, our DetCLIP-T outperforms GLIP-T by 9.9% mAP and obtains a 13.5% improvement on rare categories compared to the fully-supervised model with the same backbone as ours.