Comparison of convolutional neural networks for cloudy optical images reconstruction from single or multitemporal joint SAR and optical images

TL;DR

This paper evaluates convolutional neural networks that combine SAR and optical images to reconstruct cloud-obscured optical images, proposing a dataset creation framework and comparing single versus multiple image inputs against traditional methods.

Contribution

It introduces a dataset creation framework for training and validating CNNs for optical image reconstruction and compares single and multitemporal approaches with traditional interpolation methods.

Findings

Multitemporal CNNs outperform single-image CNNs in reconstruction quality.

The proposed dataset framework facilitates training and validation of deep learning models.

Multitemporal approaches show significant improvement over traditional interpolation.

Abstract

With the increasing availability of optical and synthetic aperture radar (SAR) images thanks to the Sentinel constellation, and the explosion of deep learning, new methods have emerged in recent years to tackle the reconstruction of optical images that are impacted by clouds. In this paper, we focus on the evaluation of convolutional neural networks that use jointly SAR and optical images to retrieve the missing contents in one single polluted optical image. We propose a simple framework that ease the creation of datasets for the training of deep nets targeting optical image reconstruction, and for the validation of machine learning based or deterministic approaches. These methods are quite different in terms of input images constraints, and comparing them is a problematic task not addressed in the literature. We show how space partitioning data structures help to query samples in terms of cloud coverage, relative acquisition date, pixel validity and relative proximity between SAR and optical images. We generate several datasets to compare the reconstructed images from networks that use a single pair of SAR and optical image, versus networks that use multiple pairs, and a traditional deterministic approach performing interpolation in temporal domain.