HFP-SAM: Hierarchical Frequency Prompted SAM for Efficient Marine Animal Segmentation

TL;DR

HFP-SAM introduces a hierarchical frequency-based framework that enhances marine animal segmentation by integrating frequency domain priors, frequency-aware region selection, and comprehensive context extraction, significantly improving performance.

Contribution

The paper proposes a novel framework combining frequency domain priors, frequency-aware region selection, and efficient context extraction to improve marine animal segmentation with SAM.

Findings

Superior performance on four public datasets

Effective integration of frequency information into SAM

Efficient extraction of spatial and channel context

Abstract

Marine Animal Segmentation (MAS) aims at identifying and segmenting marine animals from complex marine environments. Most of previous deep learning-based MAS methods struggle with the long-distance modeling issue. Recently, Segment Anything Model (SAM) has gained popularity in general image segmentation. However, it lacks of perceiving fine-grained details and frequency information. To this end, we propose a novel learning framework, named Hierarchical Frequency Prompted SAM (HFP-SAM) for high-performance MAS. First, we design a Frequency Guided Adapter (FGA) to efficiently inject marine scene information into the frozen SAM backbone through frequency domain prior masks. Additionally, we introduce a Frequency-aware Point Selection (FPS) to generate highlighted regions through frequency analysis. These regions are combined with the coarse predictions of SAM to generate point prompts and integrate into SAM's decoder for fine predictions. Finally, to obtain comprehensive segmentation masks, we introduce a Full-View Mamba (FVM) to efficiently extract spatial and channel contextual information with linear computational complexity. Extensive experiments on four public datasets demonstrate the superior performance of our approach. The source code is publicly available at https://github.com/Drchip61/TIP-HFP-SAM.