Scale-wise Bidirectional Alignment Network for Referring Remote Sensing Image Segmentation

TL;DR

This paper introduces SBANet, a novel framework for referring remote sensing image segmentation that effectively aligns visual and linguistic features across scales, improving accuracy in complex aerial imagery.

Contribution

The paper proposes a scale-wise bidirectional alignment network with learnable query tokens and dynamic feature selection for enhanced cross-modal fusion in RRSIS.

Findings

Achieves superior performance on RRSIS-D and RefSegRS datasets.

Effectively models multi-scale visual features with language guidance.

Outperforms previous state-of-the-art methods in accuracy and robustness.

Abstract

The goal of referring remote sensing image segmentation (RRSIS) is to extract specific pixel-level regions within an aerial image via a natural language expression. Recent advancements, particularly Transformer-based fusion designs, have demonstrated remarkable progress in this domain. However, existing methods primarily focus on refining visual features using language-aware guidance during the cross-modal fusion stage, neglecting the complementary vision-to-language flow. This limitation often leads to irrelevant or suboptimal representations. In addition, the diverse spatial scales of ground objects in aerial images pose significant challenges to the visual perception capabilities of existing models when conditioned on textual inputs. In this paper, we propose an innovative framework called Scale-wise Bidirectional Alignment Network (SBANet) to address these challenges for RRSIS. Specifically, we design a Bidirectional Alignment Module (BAM) with learnable query tokens to selectively and effectively represent visual and linguistic features, emphasizing regions associated with key tokens. BAM is further enhanced with a dynamic feature selection block, designed to provide both macro- and micro-level visual features, preserving global context and local details to facilitate more effective cross-modal interaction. Furthermore, SBANet incorporates a text-conditioned channel and spatial aggregator to bridge the gap between the encoder and decoder, enhancing cross-scale information exchange in complex aerial scenarios. Extensive experiments demonstrate that our proposed method achieves superior performance in comparison to previous state-of-the-art methods on the RRSIS-D and RefSegRS datasets, both quantitatively and qualitatively. The code will be released after publication.