SWIFT: A General Sensitive Weight Identification Framework for Fast Sensor-Transfer Pansharpening

TL;DR

SWIFT is a fast, unsupervised framework that efficiently adapts pansharpening models to new sensors by identifying sensitive weights, significantly reducing adaptation time while maintaining or improving performance.

Contribution

The paper introduces SWIFT, a novel unsupervised, plug-and-play method for rapid cross-sensor adaptation of pansharpening models using gradient analysis on a small informative sample set.

Findings

Reduces adaptation time from hours to about one minute.

Achieves performance comparable or superior to full retraining.

Establishes new state-of-the-art on cross-sensor pansharpening datasets.

Abstract

Pansharpening aims to fuse high-resolution panchromatic (PAN) images with low-resolution multispectral (LRMS) images to generate high-resolution multispectral (HRMS) images. Although deep learning-based methods have achieved promising performance, they generally suffer from severe performance degradation when applied to data from unseen sensors. Adapting these models through full-scale retraining or designing more complex architectures is often prohibitively expensive and impractical for real-world deployment. To address this critical challenge, we propose a fast and general-purpose framework for cross-sensor adaptation, SWIFT (Sensitive Weight Identification for Fast Transfer). Specifically, SWIFT employs an unsupervised sampling strategy based on data manifold structures to balance sample selection while mitigating the bias of traditional Farthest Point Sampling, efficiently selecting only 3\% of the most informative samples from the target domain. This subset is then used to probe a source-domain pre-trained model by analyzing the gradient behavior of its parameters, allowing for the quick identification and subsequent update of only the weight subset most sensitive to the domain shift. As a plug-and-play framework, SWIFT can be applied to various existing pansharpening models. Extensive experiments demonstrate that SWIFT reduces the adaptation time from hours to approximately one minute on a single NVIDIA RTX 4090 GPU. The adapted models not only substantially outperform direct-transfer baselines but also achieve performance competitive with, and in some cases superior to, full retraining, establishing a new state-of-the-art on cross-sensor pansharpening tasks for the WorldView-2 and QuickBird datasets.