A Simple Image Segmentation Framework via In-Context Examples

TL;DR

SINE is a straightforward image segmentation framework that uses in-context examples and a Transformer-based architecture to reduce task ambiguity and improve multi-task segmentation performance.

Contribution

The paper introduces SINE, a novel in-context segmentation method with modules for interaction and matching, enhancing task understanding and accuracy.

Findings

Effective across various segmentation tasks

Reduces task ambiguity in in-context learning

Improves segmentation accuracy with proposed modules

Abstract

Recently, there have been explorations of generalist segmentation models that can effectively tackle a variety of image segmentation tasks within a unified in-context learning framework. However, these methods still struggle with task ambiguity in in-context segmentation, as not all in-context examples can accurately convey the task information. In order to address this issue, we present SINE, a simple image Segmentation framework utilizing in-context examples. Our approach leverages a Transformer encoder-decoder structure, where the encoder provides high-quality image representations, and the decoder is designed to yield multiple task-specific output masks to effectively eliminate task ambiguity. Specifically, we introduce an In-context Interaction module to complement in-context information and produce correlations between the target image and the in-context example and a Matching Transformer that uses fixed matching and a Hungarian algorithm to eliminate differences between different tasks. In addition, we have further perfected the current evaluation system for in-context image segmentation, aiming to facilitate a holistic appraisal of these models. Experiments on various segmentation tasks show the effectiveness of the proposed method.