SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal

TL;DR

This paper introduces SEN12MS-CR-TS, a comprehensive multi-modal and multi-temporal satellite dataset designed to improve cloud removal in optical remote sensing images, and proposes two models leveraging this data for cloud-free image reconstruction.

Contribution

The paper presents a novel dataset and two models that utilize multi-modal and multi-temporal data for effective cloud removal in satellite imagery.

Findings

The dataset enhances remote sensing research capabilities.

Multi-modal and multi-temporal data improve cloud removal accuracy.

Experimental results validate the effectiveness of the proposed models.

Abstract

About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, cloud coverage affects the remote sensing practitioner's capabilities of a continuous and seamless monitoring of our planet. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing a novel multi-modal and multi-temporal data set called SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multi-modal multi-temporal 3D-Convolution Neural Network that predicts a cloud-free image from a sequence of cloudy optical and radar images. Second, a sequence-to-sequence translation model that predicts a cloud-free time series from a cloud-covered time series. Both approaches are evaluated experimentally, with their respective models trained and tested on SEN12MS-CR-TS. The conducted experiments highlight the contribution of our data set to the remote sensing community as well as the benefits of multi-modal and multi-temporal information to reconstruct noisy information. Our data set is available at https://patrickTUM.github.io/cloud_removal