Understanding What Is Not Said:Referring Remote Sensing Image Segmentation with Scarce Expressions

TL;DR

This paper introduces a weakly supervised learning paradigm for remote sensing image segmentation using limited referring expressions, supported by theoretical analysis and a novel model that improves performance with noisy, weak supervision.

Contribution

It proposes WREL, a new weakly referring expression learning framework for remote sensing, and LRB-WREL, a model that refines weak references with a learnable bank and teacher-student training.

Findings

WREL approaches fully supervised performance with limited annotations.

LRB-WREL improves robustness and generalization under noisy supervision.

Theoretical analysis confirms performance bounds of mixed-referring training.

Abstract

Referring Remote Sensing Image Segmentation (RRSIS) aims to segment instances in remote sensing images according to referring expressions. Unlike Referring Image Segmentation on general images, acquiring high-quality referring expressions in the remote sensing domain is particularly challenging due to the prevalence of small, densely distributed objects and complex backgrounds. This paper introduces a new learning paradigm, Weakly Referring Expression Learning (WREL) for RRSIS, which leverages abundant class names as weakly referring expressions together with a small set of accurate ones to enable efficient training under limited annotation conditions. Furthermore, we provide a theoretical analysis showing that mixed-referring training yields a provable upper bound on the performance gap relative to training with fully annotated referring expressions, thereby establishing the validity of this new setting. We also propose LRB-WREL, which integrates a Learnable Reference Bank (LRB) to refine weakly referring expressions through sample-specific prompt embeddings that enrich coarse class-name inputs. Combined with a teacher-student optimization framework using dynamically scheduled EMA updates, LRB-WREL stabilizes training and enhances cross-modal generalization under noisy weakly referring supervision. Extensive experiments on our newly constructed benchmark with varying weakly referring data ratios validate both the theoretical insights and the practical effectiveness of WREL and LRB-WREL, demonstrating that they can approach or even surpass models trained with fully annotated referring expressions.