Sat-JEPA-Diff: Bridging Self-Supervised Learning and Generative Diffusion for Remote Sensing

TL;DR

Sat-JEPA-Diff innovatively combines self-supervised learning with diffusion models to produce satellite images that are both structurally accurate and richly textured, overcoming limitations of traditional methods.

Contribution

It introduces a novel framework integrating SSL and diffusion models with a semantic predictor and a diffusion backbone for improved satellite image synthesis.

Findings

Achieves state-of-the-art perceptual scores on Sentinel-2 data

Outperforms deterministic baselines in boundary resolution

Maintains structural accuracy with realistic textures

Abstract

Predicting satellite imagery requires a balance between structural accuracy and textural detail. Standard deterministic methods like PredRNN or SimVP minimize pixel-based errors but suffer from the "regression to the mean" problem, producing blurry outputs that obscure subtle geographic-spatial features. Generative models provide realistic textures but often misleadingly reveal structural anomalies. To bridge this gap, we introduce Sat-JEPA-Diff, which combines Self-Supervised Learning (SSL) with Hidden Diffusion Models (LDM). An IJEPA module predicts stable semantic representations, which then route a frozen Stable Diffusion backbone via a lightweight cross-attention adapter. This ensures that the synthesized high-accuracy textures are based on absolutely accurate structural predictions. Evaluated on a global Sentinel-2 dataset, Sat-JEPA-Diff excels at resolving sharp boundaries. It achieves leading perceptual scores (GSSIM: 0.8984, FID: 0.1475) and significantly outperforms deterministic baselines, despite standard autoregressive stability limits. The code and dataset are publicly available on https://github.com/VU-AIML/SAT-JEPA-DIFF.