STARS: Shared-specific Translation and Alignment for missing-modality Remote Sensing Semantic Segmentation

TL;DR

STARS is a novel framework for remote sensing semantic segmentation that effectively handles missing modalities by using shared-specific translation, alignment mechanisms, and pixel-level semantic sampling to improve robustness and minority-class recognition.

Contribution

The paper introduces STARS, a new framework with asymmetric alignment and pixel-level semantic sampling to address missing modality challenges in remote sensing segmentation.

Findings

Reduces feature collapse with asymmetric alignment.

Improves minority-class recognition with semantic sampling.

Enhances robustness to missing modalities.

Abstract

Multimodal remote sensing technology significantly enhances the understanding of surface semantics by integrating heterogeneous data such as optical images, Synthetic Aperture Radar (SAR), and Digital Surface Models (DSM). However, in practical applications, the missing of modality data (e.g., optical or DSM) is a common and severe challenge, which leads to performance decline in traditional multimodal fusion models. Existing methods for addressing missing modalities still face limitations, including feature collapse and overly generalized recovered features. To address these issues, we propose \textbf{STARS} (\textbf{S}hared-specific \textbf{T}ranslation and \textbf{A}lignment for missing-modality \textbf{R}emote \textbf{S}ensing), a robust semantic segmentation framework for incomplete multimodal inputs. STARS is built on two key designs. First, we introduce an asymmetric alignment mechanism with bidirectional translation and stop-gradient, which effectively prevents feature collapse and reduces sensitivity to hyperparameters. Second, we propose a Pixel-level Semantic sampling Alignment (PSA) strategy that combines class-balanced pixel sampling with cross-modality semantic alignment loss, to mitigate alignment failures caused by severe class imbalance and improve minority-class recognition.