More Pictures Say More: Visual Intersection Network for Open Set Object Detection

TL;DR

This paper introduces VINO, a DETR-based model that uses a multi-image visual bank to improve open set object detection by capturing semantic intersections across visual prompts, reducing resource needs and enhancing performance.

Contribution

The paper presents a novel visual intersection network that leverages a multi-image visual bank and a visual updating mechanism for improved open set object detection, with less reliance on language modalities.

Findings

VINO achieves competitive performance on LVIS and ODinW35 benchmarks.

Requires only 7 RTX4090 GPU days per epoch, significantly reducing training resources.

Demonstrates broad applicability with a segmentation head extension.

Abstract

Open Set Object Detection has seen rapid development recently, but it continues to pose significant challenges. Language-based methods, grappling with the substantial modal disparity between textual and visual modalities, require extensive computational resources to bridge this gap. Although integrating visual prompts into these frameworks shows promise for enhancing performance, it always comes with constraints related to textual semantics. In contrast, viusal-only methods suffer from the low-quality fusion of multiple visual prompts. In response, we introduce a strong DETR-based model, Visual Intersection Network for Open Set Object Detection (VINO), which constructs a multi-image visual bank to preserve the semantic intersections of each category across all time steps. Our innovative multi-image visual updating mechanism learns to identify the semantic intersections from various visual prompts, enabling the flexible incorporation of new information and continuous optimization of feature representations. Our approach guarantees a more precise alignment between target category semantics and region semantics, while significantly reducing pre-training time and resource demands compared to language-based methods. Furthermore, the integration of a segmentation head illustrates the broad applicability of visual intersection in various visual tasks. VINO, which requires only 7 RTX4090 GPU days to complete one epoch on the Objects365v1 dataset, achieves competitive performance on par with vision-language models on benchmarks such as LVIS and ODinW35.