Improving EO Foundation Models with Confidence Assessment for enhanced Semantic segmentation

TL;DR

This paper introduces CAS, a confidence assessment model for semantic segmentation in Earth Observation, which improves accuracy by predicting the likelihood of errors at segment and pixel levels, especially for satellite data.

Contribution

The paper presents a novel confidence assessment method that enhances EO Foundation Models' semantic segmentation performance by identifying and refining incorrect predictions.

Findings

CAS outperforms baseline models in accuracy.

Effective confidence metrics for EO segmentation.

Improved land cover classification results.

Abstract

Confidence assessments of semantic segmentation algorithms are important. Ideally, deep learning models should have the ability to predict in advance whether their output is likely to be incorrect. Assessing the confidence levels of model predictions in Earth Observation (EO) classification is essential, as it can enhance semantic segmentation performance and help prevent further exploitation of the results in case of erroneous prediction. The model we developed, Confidence Assessment for enhanced Semantic segmentation (CAS), evaluates confidence at both the segment and pixel levels, providing both labels and confidence scores as output. Our model, CAS, identifies segments with incorrect predicted labels using the proposed combined confidence metric, refines the model, and enhances its performance. This work has significant applications, particularly in evaluating EO Foundation Models on semantic segmentation downstream tasks, such as land cover classification using Sentinel-2 satellite data. The evaluation results show that this strategy is effective and that the proposed model CAS outperforms other baseline models.