Generating Physically-Consistent Satellite Imagery for Climate Visualizations

TL;DR

This paper addresses the challenge of hallucinations in deep generative models for satellite imagery by introducing physics-based conditioning to improve the realism and reliability of climate-related visualizations.

Contribution

It proposes a physics-conditioned generative model for satellite imagery that reduces hallucinations and improves realism over pure deep learning approaches.

Findings

Physics-conditioned model outperforms pure deep learning model.

Model generalizes to different climate events and remote sensing data.

Published dataset includes over 30,000 labeled triplets for Earth observation.

Abstract

Deep generative vision models are now able to synthesize realistic-looking satellite imagery. But, the possibility of hallucinations prevents their adoption for risk-sensitive applications, such as generating materials for communicating climate change. To demonstrate this issue, we train a generative adversarial network (pix2pixHD) to create synthetic satellite imagery of future flooding and reforestation events. We find that a pure deep learning-based model can generate photorealistic flood visualizations but hallucinates floods at locations that were not susceptible to flooding. To address this issue, we propose to condition and evaluate generative vision models on segmentation maps of physics-based flood models. We show that our physics-conditioned model outperforms the pure deep learning-based model and a handcrafted baseline. We evaluate the generalization capability of our method to different remote sensing data and different climate-related events (reforestation). We publish our code and dataset which includes the data for a third case study of melting Arctic sea ice and $>$30,000 labeled HD image triplets -- or the equivalent of 5.5 million images at 128x128 pixels -- for segmentation guided image-to-image translation in Earth observation. Code and data is available at \url{https://github.com/blutjens/eie-earth-public}.