Visual Prompt Selection for In-Context Learning Segmentation

TL;DR

This paper improves image segmentation by developing a new context search method for visual prompts in in-context learning, leading to better performance and reduced annotation costs.

Contribution

It introduces a stepwise context search strategy that adaptively selects diverse and well-matched visual prompts, enhancing segmentation accuracy.

Findings

The proposed method outperforms existing prompt selection strategies.

Diversity of prompts significantly impacts segmentation quality.

The approach reduces annotation costs by narrowing the search space.

Abstract

As a fundamental and extensively studied task in computer vision, image segmentation aims to locate and identify different semantic concepts at the pixel level. Recently, inspired by In-Context Learning (ICL), several generalist segmentation frameworks have been proposed, providing a promising paradigm for segmenting specific objects. However, existing works mostly ignore the value of visual prompts or simply apply similarity sorting to select contextual examples. In this paper, we focus on rethinking and improving the example selection strategy. By comprehensive comparisons, we first demonstrate that ICL-based segmentation models are sensitive to different contexts. Furthermore, empirical evidence indicates that the diversity of contextual prompts plays a crucial role in guiding segmentation. Based on the above insights, we propose a new stepwise context search method. Different from previous works, we construct a small yet rich candidate pool and adaptively search the well-matched contexts. More importantly, this method effectively reduces the annotation cost by compacting the search space. Extensive experiments show that our method is an effective strategy for selecting examples and enhancing segmentation performance.