SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints

TL;DR

This paper introduces a novel framework that enhances remote sensing image segmentation by leveraging SAM's raw outputs with object and boundary constraints, improving accuracy through specialized loss functions and consistency measures.

Contribution

It proposes a new method utilizing SAM-generated object and boundary information with custom loss functions to improve semantic segmentation in remote sensing images.

Findings

Improved segmentation accuracy on ISPRS Vaihingen dataset

Enhanced boundary delineation on LoveDA Urban dataset

Effective integration of object and boundary constraints

Abstract

Semantic segmentation of remote sensing imagery plays a pivotal role in extracting precise information for diverse down-stream applications. Recent development of the Segment Anything Model (SAM), an advanced general-purpose segmentation model, has revolutionized this field, presenting new avenues for accurate and efficient segmentation. However, SAM is limited to generating segmentation results without class information. Consequently, the utilization of such a powerful general vision model for semantic segmentation in remote sensing images has become a focal point of research. In this paper, we present a streamlined framework aimed at leveraging the raw output of SAM by exploiting two novel concepts called SAM-Generated Object (SGO) and SAM-Generated Boundary (SGB). More specifically, we propose a novel object loss and further introduce a boundary loss as augmentative components to aid in model optimization in a general semantic segmentation framework. Taking into account the content characteristics of SGO, we introduce the concept of object consistency to leverage segmented regions lacking semantic information. By imposing constraints on the consistency of predicted values within objects, the object loss aims to enhance semantic segmentation performance. Furthermore, the boundary loss capitalizes on the distinctive features of SGB by directing the model's attention to the boundary information of the object. Experimental results on two well-known datasets, namely ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness of our proposed method. The source code for this work will be accessible at https://github.com/sstary/SSRS.