Leveraging Large-Scale Pretrained Spatial-Spectral Priors for General Zero-Shot Pansharpening

TL;DR

This paper introduces a pretraining strategy using large-scale simulated datasets to learn robust spatial-spectral priors, significantly enhancing the generalization of remote sensing image fusion models across diverse datasets and sensors.

Contribution

The paper proposes a novel pretraining approach leveraging simulated data to improve zero-shot and one-shot generalization in remote sensing image fusion models.

Findings

Pretrained models outperform baseline methods in zero-shot scenarios.

Significant improvement in cross-sensor generalization performance.

Effective across various network architectures including CNNs, Transformers, and Mamba.

Abstract

Existing deep learning methods for remote sensing image fusion often suffer from poor generalization when applied to unseen datasets due to the limited availability of real training data and the domain gap between different satellite sensors. To address this challenge, we explore the potential of foundation models by proposing a novel pretraining strategy that leverages large-scale simulated datasets to learn robust spatial-spectral priors. Specifically, our approach first constructs diverse simulated datasets by applying various degradation operations (blur, noise, downsampling) and augmentations (bands generation, channel shuffling, high-pass filtering, color jittering, etc.) to natural images from ImageNet and remote sensing images from SkyScript. We then pretrain fusion models on these simulated data to learn generalizable spatial-spectral representations. The pretrained models are subsequently evaluated on six datasets (WorldView-2/3/4, IKONOS, QuickBird, GaoFen-2) using zero-shot and one-shot paradigms, with both full- and freeze-tuning approaches for fine-tuning. Extensive experiments on different network architectures including convolutional neural networks, Transformer, and Mamba demonstrate that our pretraining strategy significantly improves generalization performance across different satellite sensors and imaging conditions for various fusion models. The pretrained models achieve superior results in zero-shot scenarios and show remarkable adaptation capability with minimal real data in one-shot settings. Our work provides a practical solution for cross-domain pansharpening, establishes a new benchmark for generalization in remote sensing image fusion tasks, and paves the way for leveraging foundation models through advanced training strategies.