Remote Sensing Semantic Segmentation Quality Assessment based on Vision Language Model

TL;DR

This paper introduces RS-SQA, an unsupervised model leveraging vision-language pre-trained models to assess remote sensing image segmentation quality without expert annotations, outperforming existing methods.

Contribution

The paper proposes RS-SQA, a novel unsupervised quality assessment framework using a specialized vision-language model and introduces RS-SQED, a new dataset for evaluation.

Findings

RS-SQA outperforms state-of-the-art models in segmentation quality assessment.

The model effectively differentiates various segmentation quality levels.

The dataset RS-SQED supports robust evaluation of segmentation methods.

Abstract

The complexity of scenes and variations in image quality result in significant variability in the performance of semantic segmentation methods of remote sensing imagery (RSI) in supervised real-world scenarios. This makes the evaluation of semantic segmentation quality in such scenarios an issue to be resolved. However, most of the existing evaluation metrics are developed based on expert-labeled object-level annotations, which are not applicable in such scenarios. To address this issue, we propose RS-SQA, an unsupervised quality assessment model for RSI semantic segmentation based on vision language model (VLM). This framework leverages a pre-trained RS VLM for semantic understanding and utilizes intermediate features from segmentation methods to extract implicit information about segmentation quality. Specifically, we introduce CLIP-RS, a large-scale pre-trained VLM trained with purified text to reduce textual noise and capture robust semantic information in the RS domain. Feature visualizations confirm that CLIP-RS can effectively differentiate between various levels of segmentation quality. Semantic features and low-level segmentation features are effectively integrated through a semantic-guided approach to enhance evaluation accuracy. To further support the development of RS semantic segmentation quality assessment, we present RS-SQED, a dedicated dataset sampled from four major RS semantic segmentation datasets and annotated with segmentation accuracy derived from the inference results of 8 representative segmentation methods. Experimental results on the established dataset demonstrate that RS-SQA significantly outperforms state-of-the-art quality assessment models. This provides essential support for predicting segmentation accuracy and high-quality semantic segmentation interpretation, offering substantial practical value.