Multispectral Pan-sharpening via Dual-Channel Convolutional Network with Convolutional LSTM Based Hierarchical Spatial-Spectral Feature Fusion

TL;DR

This paper introduces a dual-channel convolutional network with hierarchical spatial-spectral feature fusion using convolutional LSTM for improved multispectral pan-sharpening, effectively reducing spectral distortion and enhancing image resolution.

Contribution

The paper proposes a novel dual-channel CNN architecture with a multi-level fusion strategy and a convolutional LSTM module for better spatial-spectral feature integration in pan-sharpening.

Findings

Outperforms state-of-the-art methods on simulated and real datasets.

Effectively reduces spectral distortion in high-resolution multispectral images.

Achieves superior or competitive performance in image quality metrics.

Abstract

Multispectral pan-sharpening aims at producing a high resolution (HR) multispectral (MS) image in both spatial and spectral domains by fusing a panchromatic (PAN) image and a corresponding MS image. In this paper, we propose a novel dual-channel network (DCNet) framework for MS pan-sharpening. In our DCNet, the dual-channel backbone involves a spatial channel to capture spatial information with a 2D CNN, and a spectral channel to extract spectral information with a 3D CNN. This heterogeneous 2D/3D CNN architecture can minimize causing spectral information distortion, which typically happens in conventional 2D CNN models. In order to fully integrate the spatial and spectral features captured from different levels, we introduce a multi-level fusion strategy. Specifically, a spatial-spectral CLSTM (S$^2$-CLSTM) module is proposed for fusing the hierarchical spatial and spectral features, which can effectively capture correlations among multi-level features. The S$^2$-CLSTM module attaches two fusion ways: the intra-level fusion via bi-directional lateral connections and inter-level fusion via the cell state in the S$^2$-CLSTM. Finally, the ideal HR-MS image is recovered by a reconstruction module. Extensive experiments have been conducted at both simulated lower scale and the original scale of real-world datasets. Compared with the state-of-the-art methods, the proposed DCNet achieves superior or competitive performance.