SSFam: Scribble Supervised Salient Object Detection Family

TL;DR

This paper introduces SSFam, a novel family of scribble supervised salient object detection models leveraging the Segment Anything Model (SAM) and multi-modal inputs to improve segmentation accuracy in complex scenes.

Contribution

The paper proposes a new SSFam framework that integrates modal-aware modulators and a siamese decoder with SAM for enhanced scribble supervised salient object detection across multiple modalities.

Findings

Achieves state-of-the-art performance among scribble supervised methods.

Effectively combines RGB, depth, and thermal modalities for improved segmentation.

Close to fully supervised methods in complex scene segmentation.

Abstract

Scribble supervised salient object detection (SSSOD) constructs segmentation ability of attractive objects from surroundings under the supervision of sparse scribble labels. For the better segmentation, depth and thermal infrared modalities serve as the supplement to RGB images in the complex scenes. Existing methods specifically design various feature extraction and multi-modal fusion strategies for RGB, RGB-Depth, RGB-Thermal, and Visual-Depth-Thermal image input respectively, leading to similar model flood. As the recently proposed Segment Anything Model (SAM) possesses extraordinary segmentation and prompt interactive capability, we propose an SSSOD family based on SAM, named SSFam, for the combination input with different modalities. Firstly, different modal-aware modulators are designed to attain modal-specific knowledge which cooperates with modal-agnostic information extracted from the frozen SAM encoder for the better feature ensemble. Secondly, a siamese decoder is tailored to bridge the gap between the training with scribble prompt and the testing with no prompt for the stronger decoding ability. Our model demonstrates the remarkable performance among combinations of different modalities and refreshes the highest level of scribble supervised methods and comes close to the ones of fully supervised methods. https://github.com/liuzywen/SSFam