Open-Vocabulary Object Detection via Language Hierarchy

TL;DR

This paper introduces Language Hierarchical Self-training (LHST), a novel approach that incorporates language hierarchy into weakly-supervised object detection to improve generalization across multiple datasets.

Contribution

It presents LHST, which uses language hierarchy for label expansion and co-regularization, enhancing weakly-supervised detection and bridging vocabulary gaps.

Findings

Achieves superior generalization on 14 datasets.

Effectively mitigates image-to-box label mismatch.

Improves detection accuracy with language hierarchy integration.

Abstract

Recent studies on generalizable object detection have attracted increasing attention with additional weak supervision from large-scale datasets with image-level labels. However, weakly-supervised detection learning often suffers from image-to-box label mismatch, i.e., image-level labels do not convey precise object information. We design Language Hierarchical Self-training (LHST) that introduces language hierarchy into weakly-supervised detector training for learning more generalizable detectors. LHST expands the image-level labels with language hierarchy and enables co-regularization between the expanded labels and self-training. Specifically, the expanded labels regularize self-training by providing richer supervision and mitigating the image-to-box label mismatch, while self-training allows assessing and selecting the expanded labels according to the predicted reliability. In addition, we design language hierarchical prompt generation that introduces language hierarchy into prompt generation which helps bridge the vocabulary gaps between training and testing. Extensive experiments show that the proposed techniques achieve superior generalization performance consistently across 14 widely studied object detection datasets.