A Triple-Double Convolutional Neural Network for Panchromatic Sharpening

TL;DR

This paper introduces TDNet, a novel deep neural network with a triple-double structure and level-domain loss for pansharpening, effectively combining spatial details from panchromatic images with multispectral data to produce high-resolution multispectral images.

Contribution

The paper proposes a new triple-double neural network architecture with a level-domain loss function, integrating multi-resolution analysis and ResNet blocks for improved pansharpening performance.

Findings

Outperforms recent state-of-the-art methods on multiple datasets.

Effectively exploits spatial details from panchromatic images.

Validated through extensive experiments and ablation studies.

Abstract

Pansharpening refers to the fusion of a panchromatic image with a high spatial resolution and a multispectral image with a low spatial resolution, aiming to obtain a high spatial resolution multispectral image. In this paper, we propose a novel deep neural network architecture with level-domain based loss function for pansharpening by taking into account the following double-type structures, \emph{i.e.,} double-level, double-branch, and double-direction, called as triple-double network (TDNet). By using the structure of TDNet, the spatial details of the panchromatic image can be fully exploited and utilized to progressively inject into the low spatial resolution multispectral image, thus yielding the high spatial resolution output. The specific network design is motivated by the physical formula of the traditional multi-resolution analysis (MRA) methods. Hence, an effective MRA fusion module is also integrated into the TDNet. Besides, we adopt a few ResNet blocks and some multi-scale convolution kernels to deepen and widen the network to effectively enhance the feature extraction and the robustness of the proposed TDNet. Extensive experiments on reduced- and full-resolution datasets acquired by WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate the superiority of the proposed TDNet compared with some recent state-of-the-art pansharpening approaches. An ablation study has also corroborated the effectiveness of the proposed approach.