OW-CLIP: Data-Efficient Visual Supervision for Open-World Object Detection via Human-AI Collaboration

TL;DR

OW-CLIP introduces a data-efficient, human-AI collaborative system for open-world object detection, reducing data needs and improving adaptability through novel prompt tuning, data refinement, and visualization tools.

Contribution

The paper presents OW-CLIP, a modular system combining prompt tuning, data refinement, and visualization to enable efficient open-world object detection with minimal data.

Findings

Achieves 89% of SOTA performance with only 3.8% self-generated data.

Outperforms SOTA when trained with equivalent data volumes.

Effective visualization improves annotation quality.

Abstract

Open-world object detection (OWOD) extends traditional object detection to identifying both known and unknown object, necessitating continuous model adaptation as new annotations emerge. Current approaches face significant limitations: 1) data-hungry training due to reliance on a large number of crowdsourced annotations, 2) susceptibility to "partial feature overfitting," and 3) limited flexibility due to required model architecture modifications. To tackle these issues, we present OW-CLIP, a visual analytics system that provides curated data and enables data-efficient OWOD model incremental training. OW-CLIP implements plug-and-play multimodal prompt tuning tailored for OWOD settings and introduces a novel "Crop-Smoothing" technique to mitigate partial feature overfitting. To meet the data requirements for the training methodology, we propose dual-modal data refinement methods that leverage large language models and cross-modal similarity for data generation and filtering. Simultaneously, we develope a visualization interface that enables users to explore and deliver high-quality annotations: including class-specific visual feature phrases and fine-grained differentiated images. Quantitative evaluation demonstrates that OW-CLIP achieves competitive performance at 89% of state-of-the-art performance while requiring only 3.8% self-generated data, while outperforming SOTA approach when trained with equivalent data volumes. A case study shows the effectiveness of the developed method and the improved annotation quality of our visualization system.