Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery

TL;DR

This paper introduces a large, diverse dataset of co-registered radar and optical satellite images for global, all-season cloud removal, and proposes a novel model that generalizes well across varying cloud coverage conditions.

Contribution

It provides the first publicly available global dataset of real radar and optical images for cloud removal and introduces a new model capable of handling extreme cloud coverage scenarios.

Findings

The proposed model outperforms existing methods on the new dataset.

Training on real data yields better results than synthetic data.

The dataset enables evaluation of image quality and diversity in cloud removal.

Abstract

This work has been accepted by IEEE TGRS for publication. The majority of optical observations acquired via spaceborne earth imagery are affected by clouds. While there is numerous prior work on reconstructing cloud-covered information, previous studies are oftentimes confined to narrowly-defined regions of interest, raising the question of whether an approach can generalize to a diverse set of observations acquired at variable cloud coverage or in different regions and seasons. We target the challenge of generalization by curating a large novel data set for training new cloud removal approaches and evaluate on two recently proposed performance metrics of image quality and diversity. Our data set is the first publically available to contain a global sample of co-registered radar and optical observations, cloudy as well as cloud-free. Based on the observation that cloud coverage varies widely between clear skies and absolute coverage, we propose a novel model that can deal with either extremes and evaluate its performance on our proposed data set. Finally, we demonstrate the superiority of training models on real over synthetic data, underlining the need for a carefully curated data set of real observations. To facilitate future research, our data set is made available online