Convolutional LSTMs for Cloud-Robust Segmentation of Remote Sensing Imagery

TL;DR

This paper demonstrates that convolutional LSTM networks can inherently learn to filter clouds in satellite imagery, reducing the need for complex pre-processing in cloud-robust remote sensing segmentation.

Contribution

It shows that convolutional LSTMs can internalize cloud-filtering mechanisms without explicit cloud labels, challenging the need for pre-processing pipelines.

Findings

Network cells close gates on cloudy pixels, indicating learned cloud filtering.

Achieved state-of-the-art vegetation classification accuracy without explicit cloud filtering.

Visualizations reveal internal cloud-robust features in the LSTM network.

Abstract

Clouds frequently cover the Earth's surface and pose an omnipresent challenge to optical Earth observation methods. The vast majority of remote sensing approaches either selectively choose single cloud-free observations or employ a pre-classification strategy to identify and mask cloudy pixels. We follow a different strategy and treat cloud coverage as noise that is inherent to the observed satellite data. In prior work, we directly employed a straightforward \emph{convolutional long short-term memory} network for vegetation classification without explicit cloud filtering and achieved state-of-the-art classification accuracies. In this work, we investigate this cloud-robustness further by visualizing internal cell activations and performing an ablation experiment on datasets of different cloud coverage. In the visualizations of network states, we identified some cells in which modulation and input gates closed on cloudy pixels. This indicates that the network has internalized a cloud-filtering mechanism without being specifically trained on cloud labels. Overall, our results question the necessity of sophisticated pre-processing pipelines for multi-temporal deep learning approaches.