Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm

TL;DR

This paper introduces the Image Prompt Paradigm for open-set visual perception tasks, enabling detection and segmentation of specialized categories using only a few image prompts without human interaction, and demonstrates competitive performance.

Contribution

It proposes a novel image prompt paradigm and a framework called MI Grounding that automatically encodes, selects, and fuses image prompts for non-interactive open-set detection and segmentation.

Findings

Achieves competitive results on public OSOD and OSS benchmarks.

Outperforms existing methods on the ADR50K dataset.

Enables fully automated detection and segmentation with minimal prompts.

Abstract

To break through the limitations of pre-training models on fixed categories, Open-Set Object Detection (OSOD) and Open-Set Segmentation (OSS) have attracted a surge of interest from researchers. Inspired by large language models, mainstream OSOD and OSS methods generally utilize text as a prompt, achieving remarkable performance. Following SAM paradigm, some researchers use visual prompts, such as points, boxes, and masks that cover detection or segmentation targets. Despite these two prompt paradigms exhibit excellent performance, they also reveal inherent limitations. On the one hand, it is difficult to accurately describe characteristics of specialized category using textual description. On the other hand, existing visual prompt paradigms heavily rely on multi-round human interaction, which hinders them being applied to fully automated pipeline. To address the above issues, we propose a novel prompt paradigm in OSOD and OSS, that is, \textbf{Image Prompt Paradigm}. This brand new prompt paradigm enables to detect or segment specialized categories without multi-round human intervention. To achieve this goal, the proposed image prompt paradigm uses just a few image instances as prompts, and we propose a novel framework named \textbf{MI Grounding} for this new paradigm. In this framework, high-quality image prompts are automatically encoded, selected and fused, achieving the single-stage and non-interactive inference. We conduct extensive experiments on public datasets, showing that MI Grounding achieves competitive performance on OSOD and OSS benchmarks compared to text prompt paradigm methods and visual prompt paradigm methods. Moreover, MI Grounding can greatly outperform existing method on our constructed specialized ADR50K dataset.