VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion

TL;DR

VL-SAM-V2 is an innovative open-world object detection framework that fuses open-set and open-ended queries, enabling discovery of unseen objects with improved performance, especially on rare categories.

Contribution

The paper introduces a novel query fusion module and ranked learnable queries for open-world detection, enhancing the ability to discover unseen objects without human input.

Findings

Outperforms previous open-set and open-ended methods on LVIS.

Excels particularly on rare object categories.

Demonstrates flexible evaluation in open-set and open-ended modes.

Abstract

Current perception models have achieved remarkable success by leveraging large-scale labeled datasets, but still face challenges in open-world environments with novel objects. To address this limitation, researchers introduce open-set perception models to detect or segment arbitrary test-time user-input categories. However, open-set models rely on human involvement to provide predefined object categories as input during inference. More recently, researchers have framed a more realistic and challenging task known as open-ended perception that aims to discover unseen objects without requiring any category-level input from humans at inference time. Nevertheless, open-ended models suffer from low performance compared to open-set models. In this paper, we present VL-SAM-V2, an open-world object detection framework that is capable of discovering unseen objects while achieving favorable performance. To achieve this, we combine queries from open-set and open-ended models and propose a general and specific query fusion module to allow different queries to interact. By adjusting queries from open-set models, we enable VL-SAM-V2 to be evaluated in the open-set or open-ended mode. In addition, to learn more diverse queries, we introduce ranked learnable queries to match queries with proposals from open-ended models by sorting. Moreover, we design a denoising point training strategy to facilitate the training process. Experimental results on LVIS show that our method surpasses the previous open-set and open-ended methods, especially on rare objects.