CloudBreaker: Breaking the Cloud Covers of Sentinel-2 Images using Multi-Stage Trained Conditional Flow Matching on Sentinel-1

TL;DR

CloudBreaker is a novel framework that synthesizes high-quality multi-spectral Sentinel-2 images from Sentinel-1 radar data, overcoming cloud cover limitations and enabling reliable remote sensing under various conditions.

Contribution

It introduces a multi-stage training approach based on conditional flow matching with cosine scheduling, pioneering the integration of these techniques for satellite image synthesis.

Findings

Achieved a FID score of 0.7432 indicating high image realism

Attained SSIM scores of 0.6156 for NDWI and 0.6874 for NDVI

Demonstrated effective generation of optical images from radar data

Abstract

Cloud cover and nighttime conditions remain significant limitations in satellite-based remote sensing, often restricting the availability and usability of multi-spectral imagery. In contrast, Sentinel-1 radar images are unaffected by cloud cover and can provide consistent data regardless of weather or lighting conditions. To address the challenges of limited satellite imagery, we propose CloudBreaker, a novel framework that generates high-quality multi-spectral Sentinel-2 signals from Sentinel-1 data. This includes the reconstruction of optical (RGB) images as well as critical vegetation and water indices such as NDVI and NDWI. We employed a novel multi-stage training approach based on conditional latent flow matching and, to the best of our knowledge, are the first to integrate cosine scheduling with flow matching. CloudBreaker demonstrates strong performance, achieving a Frechet Inception Distance (FID) score of 0.7432, indicating high fidelity and realism in the generated optical imagery. The model also achieved Structural Similarity Index Measure (SSIM) of 0.6156 for NDWI and 0.6874 for NDVI, indicating a high degree of structural similarity. This establishes CloudBreaker as a promising solution for a wide range of remote sensing applications where multi-spectral data is typically unavailable or unreliable