SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation

TL;DR

This paper introduces SPDA-SAM, a novel instance segmentation model that enhances SAM with self-generated prompts and depth information fusion, significantly improving performance across multiple datasets.

Contribution

The paper proposes a self-prompted, depth-aware extension of SAM with novel modules for semantic-spatial prompting and RGB-D feature fusion, addressing limitations of manual prompts and lack of depth data.

Findings

Outperforms state-of-the-art methods on twelve datasets.

Effective use of depth maps improves spatial understanding.

Self-prompting reduces dependency on manual prompts.

Abstract

Recently, Segment Anything Model (SAM) has demonstrated strong generalizability in various instance segmentation tasks. However, its performance is severely dependent on the quality of manual prompts. In addition, the RGB images that instance segmentation methods normally use inherently lack depth information. As a result, the ability of these methods to perceive spatial structures and delineate object boundaries is hindered. To address these challenges, we propose a Self-prompted Depth-Aware SAM (SPDA-SAM) for instance segmentation. Specifically, we design a Semantic-Spatial Self-prompt Module (SSSPM) which extracts the semantic and spatial prompts from the image encoder and the mask decoder of SAM, respectively. Furthermore, we introduce a Coarse-to-Fine RGB-D Fusion Module (C2FFM), in which the features extracted from a monocular RGB image and the depth map estimated from it are fused. In particular, the structural information in the depth map is used to provide coarse-grained guidance to feature fusion, while local variations in depth are encoded in order to fuse fine-grained feature representations. To our knowledge, SAM has not been explored in such self-prompted and depth-aware manners. Experimental results demonstrate that our SPDA-SAM outperforms its state-of-the-art counterparts across twelve different data sets. These promising results should be due to the guidance of the self-prompts and the compensation for the spatial information loss by the coarse-to-fine RGB-D fusion operation.