Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation

TL;DR

This paper introduces SeeCo, a plug-and-play framework that dynamically recalibrates open-vocabulary remote sensing segmentation models during inference by leveraging geometric and semantic consensus, improving accuracy across diverse scenes.

Contribution

The paper presents a novel on-the-fly recalibration method for OVSS models that does not require retraining, enhancing scene-specific semantic-geometric alignment during inference.

Findings

Consistent performance improvements across eight benchmarks.

Effective alleviation of semantic bias and under-activation issues.

Universal applicability to different OVSS models without retraining.

Abstract

Open-vocabulary semantic segmentation (OVSS) in remote sensing images is a promising task that employs textual descriptions for identifying undefined land cover categories. Despite notable advances, existing methods typically employ a static inference paradigm, overlooking the distinct distribution of each scene, resulting in semantic ambiguity in diverse land covers and incomplete foreground activation. Motivated by this, we propose Seeking Consensus, termed SeeCo, a plug-and-play framework to boost the performance of training-free OVSS models in remote sensing images, which recalibrates arbitrary OVSS models on-the-fly by seeking dual consensus: geometric consensus learning (GCL) through multi-view consistent observations and semantic consensus learning (SCL) via textual description adaptive calibration, which assists collaborative recalibration of visual and textual semantics. The two consensus are injected via an online consensus injector (OCI), effectively alleviating the under-activation and semantic bias. SeeCo requires no specific training process, yet recalibrates semantic-geometric alignment for each unique scene during inference. Extensive experiments on eight remote sensing OVSS benchmarks show consistent gains, proving its effectiveness and universality.