SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model

TL;DR

SAM-REF enhances interactive segmentation by combining the accuracy of early fusion with the efficiency of late fusion through a two-stage refinement, improving detail extraction without increasing computational cost.

Contribution

Introduces SAM-REF, a two-stage refinement framework that integrates image and prompt information effectively, surpassing existing methods in segmentation quality while maintaining efficiency.

Findings

Outperforms state-of-the-art in segmentation metrics

Maintains high efficiency comparable to late fusion models

Improves detail extraction in interactive segmentation

Abstract

Interactive segmentation is to segment the mask of the target object according to the user's interactive prompts. There are two mainstream strategies: early fusion and late fusion. Current specialist models utilize the early fusion strategy that encodes the combination of images and prompts to target the prompted objects, yet repetitive complex computations on the images result in high latency. Late fusion models extract image embeddings once and merge them with the prompts in later interactions. This strategy avoids redundant image feature extraction and improves efficiency significantly. A recent milestone is the Segment Anything Model (SAM). However, this strategy limits the models' ability to extract detailed information from the prompted target zone. To address this issue, we propose SAM-REF, a two-stage refinement framework that fully integrates images and prompts by using a lightweight refiner into the interaction of late fusion, which combines the accuracy of early fusion and maintains the efficiency of late fusion. Through extensive experiments, we show that our SAM-REF model outperforms the current state-of-the-art method in most metrics on segmentation quality without compromising efficiency.