PanFlowNet: A Flow-Based Deep Network for Pan-sharpening

TL;DR

PanFlowNet introduces a flow-based deep learning model for pan-sharpening that captures the distribution of possible high-resolution multispectral images, enabling the generation of diverse, high-quality HRMS images from LRMS and PAN inputs.

Contribution

It presents a novel invertible network architecture that models the conditional distribution of HRMS images, addressing the ill-posed nature of pan-sharpening and allowing for diverse outputs.

Findings

Generates diverse HRMS images from the same LRMS and PAN inputs.

Outperforms state-of-the-art methods in visual and quantitative evaluations.

Effective on various satellite datasets.

Abstract

Pan-sharpening aims to generate a high-resolution multispectral (HRMS) image by integrating the spectral information of a low-resolution multispectral (LRMS) image with the texture details of a high-resolution panchromatic (PAN) image. It essentially inherits the ill-posed nature of the super-resolution (SR) task that diverse HRMS images can degrade into an LRMS image. However, existing deep learning-based methods recover only one HRMS image from the LRMS image and PAN image using a deterministic mapping, thus ignoring the diversity of the HRMS image. In this paper, to alleviate this ill-posed issue, we propose a flow-based pan-sharpening network (PanFlowNet) to directly learn the conditional distribution of HRMS image given LRMS image and PAN image instead of learning a deterministic mapping. Specifically, we first transform this unknown conditional distribution into a given Gaussian distribution by an invertible network, and the conditional distribution can thus be explicitly defined. Then, we design an invertible Conditional Affine Coupling Block (CACB) and further build the architecture of PanFlowNet by stacking a series of CACBs. Finally, the PanFlowNet is trained by maximizing the log-likelihood of the conditional distribution given a training set and can then be used to predict diverse HRMS images. The experimental results verify that the proposed PanFlowNet can generate various HRMS images given an LRMS image and a PAN image. Additionally, the experimental results on different kinds of satellite datasets also demonstrate the superiority of our PanFlowNet compared with other state-of-the-art methods both visually and quantitatively.