DepthSeg: Depth prompting in remote sensing semantic segmentation

TL;DR

DepthSeg introduces a novel framework that incorporates depth information into remote sensing semantic segmentation, improving accuracy in complex scenarios like shadow occlusion and spectral confusion by explicitly modeling and integrating height data.

Contribution

The paper presents a depth prompting framework that models and utilizes depth/height information in 2D remote sensing images, enhancing segmentation accuracy over existing methods.

Findings

DepthSeg outperforms baseline models on LiuZhou dataset.

Depth prompts significantly improve land cover classification accuracy.

Ablation studies confirm the effectiveness of depth prompting in complex scenes.

Abstract

Remote sensing semantic segmentation is crucial for extracting detailed land surface information, enabling applications such as environmental monitoring, land use planning, and resource assessment. In recent years, advancements in artificial intelligence have spurred the development of automatic remote sensing semantic segmentation methods. However, the existing semantic segmentation methods focus on distinguishing spectral characteristics of different objects while ignoring the differences in the elevation of the different targets. This results in land cover misclassification in complex scenarios involving shadow occlusion and spectral confusion. In this paper, we introduce a depth prompting two-dimensional (2D) remote sensing semantic segmentation framework (DepthSeg). It automatically models depth/height information from 2D remote sensing images and integrates it into the semantic segmentation framework to mitigate the effects of spectral confusion and shadow occlusion. During the feature extraction phase of DepthSeg, we introduce a lightweight adapter to enable cost-effective fine-tuning of the large-parameter vision transformer encoder pre-trained by natural images. In the depth prompting phase, we propose a depth prompter to model depth/height features explicitly. In the semantic prediction phase, we introduce a semantic classification decoder that couples the depth prompts with high-dimensional land-cover features, enabling accurate extraction of land-cover types. Experiments on the LiuZhou dataset validate the advantages of the DepthSeg framework in land cover mapping tasks. Detailed ablation studies further highlight the significance of the depth prompts in remote sensing semantic segmentation.