Taming Self-Training for Open-Vocabulary Object Detection

TL;DR

This paper introduces SAS-Det, a novel approach for open-vocabulary object detection that addresses challenges of noisy pseudo labels and distribution shifts by splitting detection heads and stabilizing teacher updates, leading to improved performance.

Contribution

SAS-Det proposes a split-and-fusion detection head and a periodic teacher update strategy to enhance self-training for open-vocabulary detection, reducing noise and stabilizing training.

Findings

Outperforms recent models on COCO and LVIS benchmarks.

Achieves 37.4 AP50 and 29.1 APr on novel categories.

Demonstrates efficiency and effectiveness of the proposed methods.

Abstract

Recent studies have shown promising performance in open-vocabulary object detection (OVD) by utilizing pseudo labels (PLs) from pretrained vision and language models (VLMs). However, teacher-student self-training, a powerful and widely used paradigm to leverage PLs, is rarely explored for OVD. This work identifies two challenges of using self-training in OVD: noisy PLs from VLMs and frequent distribution changes of PLs. To address these challenges, we propose SAS-Det that tames self-training for OVD from two key perspectives. First, we present a split-and-fusion (SAF) head that splits a standard detection into an open-branch and a closed-branch. This design can reduce noisy supervision from pseudo boxes. Moreover, the two branches learn complementary knowledge from different training data, significantly enhancing performance when fused together. Second, in our view, unlike in closed-set tasks, the PL distributions in OVD are solely determined by the teacher model. We introduce a periodic update strategy to decrease the number of updates to the teacher, thereby decreasing the frequency of changes in PL distributions, which stabilizes the training process. Extensive experiments demonstrate SAS-Det is both efficient and effective. SAS-Det outperforms recent models of the same scale by a clear margin and achieves 37.4 AP50 and 29.1 APr on novel categories of the COCO and LVIS benchmarks, respectively. Code is available at \url{https://github.com/xiaofeng94/SAS-Det}.