Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM

TL;DR

This paper introduces Dual-SAM, a novel framework that enhances marine animal segmentation by integrating multi-level features, prior information, and pixel connectivity, achieving state-of-the-art results in underwater image analysis.

Contribution

The paper proposes Dual-SAM, a new marine animal segmentation method that combines a dual structure, multi-level prompts, and a connectivity prediction paradigm for improved accuracy.

Findings

Achieves state-of-the-art performance on five MAS datasets.

Effectively captures inter-connectivity between marine animals.

Enhances feature learning with multi-level prompts and adapters.

Abstract

As an important pillar of underwater intelligence, Marine Animal Segmentation (MAS) involves segmenting animals within marine environments. Previous methods don't excel in extracting long-range contextual features and overlook the connectivity between discrete pixels. Recently, Segment Anything Model (SAM) offers a universal framework for general segmentation tasks. Unfortunately, trained with natural images, SAM does not obtain the prior knowledge from marine images. In addition, the single-position prompt of SAM is very insufficient for prior guidance. To address these issues, we propose a novel feature learning framework, named Dual-SAM for high-performance MAS. To this end, we first introduce a dual structure with SAM's paradigm to enhance feature learning of marine images. Then, we propose a Multi-level Coupled Prompt (MCP) strategy to instruct comprehensive underwater prior information, and enhance the multi-level features of SAM's encoder with adapters. Subsequently, we design a Dilated Fusion Attention Module (DFAM) to progressively integrate multi-level features from SAM's encoder. Finally, instead of directly predicting the masks of marine animals, we propose a Criss-Cross Connectivity Prediction (C$^3$P) paradigm to capture the inter-connectivity between discrete pixels. With dual decoders, it generates pseudo-labels and achieves mutual supervision for complementary feature representations, resulting in considerable improvements over previous techniques. Extensive experiments verify that our proposed method achieves state-of-the-art performances on five widely-used MAS datasets. The code is available at https://github.com/Drchip61/Dual_SAM.