PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model

TL;DR

PolSAM is an advanced segmentation model that incorporates polarimetric scattering mechanisms and novel prompts to improve PolSAR data analysis, outperforming existing models in accuracy and efficiency.

Contribution

The paper introduces PolSAM, a novel SAM-based model that integrates domain-specific scattering features and new prompt strategies for enhanced PolSAR segmentation.

Findings

PolSAM outperforms existing models on PhySAR-Seg datasets.

PolSAM reduces data storage and speeds up inference.

PolSAM achieves higher segmentation accuracy.

Abstract

PolSAR data presents unique challenges due to its rich and complex characteristics. Existing data representations, such as complex-valued data, polarimetric features, and amplitude images, are widely used. However, these formats often face issues related to usability, interpretability, and data integrity. Most feature extraction networks for PolSAR are small, limiting their ability to capture features effectively. To address these issues, We propose the Polarimetric Scattering Mechanism-Informed SAM (PolSAM), an enhanced Segment Anything Model (SAM) that integrates domain-specific scattering characteristics and a novel prompt generation strategy. PolSAM introduces Microwave Vision Data (MVD), a lightweight and interpretable data representation derived from polarimetric decomposition and semantic correlations. We propose two key components: the Feature-Level Fusion Prompt (FFP), which fuses visual tokens from pseudo-colored SAR images and MVD to address modality incompatibility in the frozen SAM encoder, and the Semantic-Level Fusion Prompt (SFP), which refines sparse and dense segmentation prompts using semantic information. Experimental results on the PhySAR-Seg datasets demonstrate that PolSAM significantly outperforms existing SAM-based and multimodal fusion models, improving segmentation accuracy, reducing data storage, and accelerating inference time. The source code and datasets will be made publicly available at https://github.com/XAI4SAR/PolSAM.