Unsupervised Cycle-consistent Generative Adversarial Networks for Pan-sharpening

TL;DR

This paper introduces an unsupervised cycle-consistent GAN framework for pan-sharpening that learns directly from full-scale images without ground truths, improving practical applicability and performance over existing supervised methods.

Contribution

It proposes a novel unsupervised GAN approach with a hybrid loss for pan-sharpening, overcoming scale gap issues in supervised methods.

Findings

Significantly improves pan-sharpening on full-scale images

Outperforms state-of-the-art methods on GaoFen-2 and WorldView-3 datasets

Demonstrates practical value for real-world satellite image processing

Abstract

Deep learning based pan-sharpening has received significant research interest in recent years. Most of existing methods fall into the supervised learning framework in which they down-sample the multi-spectral (MS) and panchromatic (PAN) images and regard the original MS images as ground truths to form training samples. Although impressive performance could be achieved, they have difficulties generalizing to the original full-scale images due to the scale gap, which makes them lack of practicability. In this paper, we propose an unsupervised generative adversarial framework that learns from the full-scale images without the ground truths to alleviate this problem. We extract the modality-specific features from the PAN and MS images with a two-stream generator, perform fusion in the feature domain, and then reconstruct the pan-sharpened images. Furthermore, we introduce a novel hybrid loss based on the cycle-consistency and adversarial scheme to improve the performance. Comparison experiments with the state-of-the-art methods are conducted on GaoFen-2 and WorldView-3 satellites. Results demonstrate that the proposed method can greatly improve the pan-sharpening performance on the full-scale images, which clearly show its practical value. Codes are available at https://github.com/zhysora/UCGAN.