U2Net: A General Framework with Spatial-Spectral-Integrated Double U-Net for Image Fusion

TL;DR

U2Net is a novel double U-shaped network that integrates spatial and spectral features for improved image fusion, outperforming state-of-the-art methods in remote sensing and hyperspectral image tasks.

Contribution

The paper introduces a new spatial-spectral integration structure called S2Block within a double U-Net framework for enhanced image fusion performance.

Findings

U2Net achieves superior quantitative results over existing methods.

U2Net demonstrates better qualitative image quality in experiments.

The proposed S2Block effectively combines multi-source features.

Abstract

In image fusion tasks, images obtained from different sources exhibit distinct properties. Consequently, treating them uniformly with a single-branch network can lead to inadequate feature extraction. Additionally, numerous works have demonstrated that multi-scaled networks capture information more sufficiently than single-scaled models in pixel-level computer vision problems. Considering these factors, we propose U2Net, a spatial-spectral-integrated double U-shape network for image fusion. The U2Net utilizes a spatial U-Net and a spectral U-Net to extract spatial details and spectral characteristics, which allows for the discriminative and hierarchical learning of features from diverse images. In contrast to most previous works that merely employ concatenation to merge spatial and spectral information, this paper introduces a novel spatial-spectral integration structure called S2Block, which combines feature maps from different sources in a logical and effective way. We conduct a series of experiments on two image fusion tasks, including remote sensing pansharpening and hyperspectral image super-resolution (HISR). The U2Net outperforms representative state-of-the-art (SOTA) approaches in both quantitative and qualitative evaluations, demonstrating the superiority of our method. The code is available at https://github.com/PSRben/U2Net.