Vision Transformer-Based Time-Series Image Reconstruction for Cloud-Filling Applications

TL;DR

This paper introduces a Vision Transformer-based framework for reconstructing multispectral images in cloud-covered regions by leveraging temporal coherence and SAR data, significantly improving reconstruction accuracy.

Contribution

The novel ViT-based framework effectively combines MSI and SAR data over time for improved cloud-covered MSI reconstruction, outperforming existing methods.

Findings

Significantly better reconstruction accuracy than baseline methods.

Effective utilization of temporal coherence and SAR data.

Robust performance demonstrated through comprehensive experiments.

Abstract

Cloud cover in multispectral imagery (MSI) poses significant challenges for early season crop mapping, as it leads to missing or corrupted spectral information. Synthetic aperture radar (SAR) data, which is not affected by cloud interference, offers a complementary solution, but lack sufficient spectral detail for precise crop mapping. To address this, we propose a novel framework, Time-series MSI Image Reconstruction using Vision Transformer (ViT), to reconstruct MSI data in cloud-covered regions by leveraging the temporal coherence of MSI and the complementary information from SAR from the attention mechanism. Comprehensive experiments, using rigorous reconstruction evaluation metrics, demonstrate that Time-series ViT framework significantly outperforms baselines that use non-time-series MSI and SAR or time-series MSI without SAR, effectively enhancing MSI image reconstruction in cloud-covered regions.