Learning from Exemplars for Interactive Image Segmentation

TL;DR

This paper introduces novel transformer-based interactive image segmentation models that leverage exemplar information to efficiently segment multiple objects of the same category, reducing user effort and improving accuracy.

Contribution

The paper proposes new frameworks for interactive segmentation that utilize exemplar-informed modules and cross-attention to enhance multi-object segmentation performance.

Findings

Achieves superior performance on benchmark datasets.

Reduces user clicks by around 15%.

Requires two fewer clicks to reach high IoU thresholds.

Abstract

Interactive image segmentation enables users to interact minimally with a machine, facilitating the gradual refinement of the segmentation mask for a target of interest. Previous studies have demonstrated impressive performance in extracting a single target mask through interactive segmentation. However, the information cues of previously interacted objects have been overlooked in the existing methods, which can be further explored to speed up interactive segmentation for multiple targets in the same category. To this end, we introduce novel interactive segmentation frameworks for both a single object and multiple objects in the same category. Specifically, our model leverages transformer backbones to extract interaction-focused visual features from the image and the interactions to obtain a satisfactory mask of a target as an exemplar. For multiple objects, we propose an exemplar-informed module to enhance the learning of similarities among the objects of the target category. To combine attended features from different modules, we incorporate cross-attention blocks followed by a feature fusion module. Experiments conducted on mainstream benchmarks demonstrate that our models achieve superior performance compared to previous methods. Particularly, our model reduces users' labor by around 15\%, requiring two fewer clicks to achieve target IoUs 85\% and 90\%. The results highlight our models' potential as a flexible and practical annotation tool. The source code will be released after publication.