CBEN -- A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understanding

TL;DR

This paper introduces the CBEN dataset, combining optical and radar satellite images with cloud occlusion, to develop and evaluate cloud-robust remote sensing methods, demonstrating significant performance improvements.

Contribution

The creation of the CBEN dataset with cloudy optical and radar images and the evaluation of methods adapted for cloud robustness are novel contributions.

Findings

State-of-the-art methods drop 23-33% AP on cloudy images.

Training with cloudy optical data improves performance by 17.2-28.7%.

Code and dataset are publicly available at the provided GitHub link.

Abstract

Clouds are a common phenomenon that distorts optical satellite imagery, which poses a challenge for remote sensing. However, in the literature cloudless analysis is often performed where cloudy images are excluded from machine learning datasets and methods. Such an approach cannot be applied to time sensitive applications, e.g., during natural disasters. A possible solution is to apply cloud removal as a preprocessing step to ensure that cloudfree solutions are not failing under such conditions. But cloud removal methods are still actively researched and suffer from drawbacks, such as generated visual artifacts. Therefore, it is desirable to develop cloud robust methods that are less affected by cloudy weather. Cloud robust methods can be achieved by combining optical data with radar, a modality unaffected by clouds. While many datasets for machine learning combine optical and radar data, most researchers exclude cloudy images. We identify this exclusion from machine learning training and evaluation as a limitation that reduces applicability to cloudy scenarios. To investigate this, we assembled a dataset, named CloudyBigEarthNet (CBEN), of paired optical and radar images with cloud occlusion for training and evaluation. Using average precision (AP) as the evaluation metric, we show that state-of-the-art methods trained on combined clear-sky optical and radar imagery suffer performance drops of 23-33 percentage points when evaluated on cloudy images. We then adapt these methods to cloudy optical data during training, achieving relative improvement of 17.2-28.7 percentage points on cloudy test cases compared with the original approaches. Code and dataset are publicly available at: https://github.com/mstricker13/CBEN