DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection

TL;DR

DST-Det introduces a simple, efficient self-training approach leveraging pre-trained vision-language models to improve open-vocabulary object detection, enhancing recall and accuracy for novel classes without extra annotations or re-training.

Contribution

The paper proposes a novel self-training strategy that selects proposals as background or novel classes, improving open-vocabulary detection without additional data or re-training.

Findings

Significant performance improvements on LVIS, V3Det, and COCO datasets.

Achieved 1.7% better AP on LVIS compared to F-VLM.

Reaches 46.7 novel class AP on COCO without extra data.

Abstract

Open-vocabulary object detection (OVOD) aims to detect the objects beyond the set of classes observed during training. This work introduces a straightforward and efficient strategy that utilizes pre-trained vision-language models (VLM), like CLIP, to identify potential novel classes through zero-shot classification. Previous methods use a class-agnostic region proposal network to detect object proposals and consider the proposals that do not match the ground truth as background. Unlike these methods, our method will select a subset of proposals that will be considered as background during the training. Then, we treat them as novel classes during training. We refer to this approach as the self-training strategy, which enhances recall and accuracy for novel classes without requiring extra annotations, datasets, and re-training. Compared to previous pseudo methods, our approach does not require re-training and offline labeling processing, which is more efficient and effective in one-shot training. Empirical evaluations on three datasets, including LVIS, V3Det, and COCO, demonstrate significant improvements over the baseline performance without incurring additional parameters or computational costs during inference. In addition, we also apply our method to various baselines. In particular, compared with the previous method, F-VLM, our method achieves a 1.7% improvement on the LVIS dataset. Combined with the recent method CLIPSelf, our method also achieves 46.7 novel class AP on COCO without introducing extra data for pertaining. We also achieve over 6.5% improvement over the F-VLM baseline in the recent challenging V3Det dataset. We release our code and models at https://github.com/xushilin1/dst-det.