Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening

TL;DR

This paper introduces a novel deep unfolded network using multi-head attention residual blocks for satellite image fusion, combining model-based optimization with deep learning for improved pansharpening and hypersharpening performance.

Contribution

It proposes a model-based deep unfolded approach with a multi-head attention residual network that leverages geometric information and self-similarities for enhanced image fusion.

Findings

Outperforms existing methods on multiple datasets

Demonstrates strong generalization across sensors and resolutions

Enhances fused image quality with residual attention mechanisms

Abstract

The objective of pansharpening and hypersharpening is to accurately combine a high-resolution panchromatic (PAN) image with a low-resolution multispectral (MS) or hyperspectral (HS) image, respectively. Unfolding fusion methods integrate the powerful representation capabilities of deep learning with the robustness of model-based approaches. These techniques involve unrolling the steps of the optimization scheme derived from the minimization of an energy into a deep learning framework, resulting in efficient and highly interpretable architectures. In this paper, we propose a model-based deep unfolded method for satellite image fusion. Our approach is based on a variational formulation that incorporates the classic observation model for MS/HS data, a high-frequency injection constraint based on the PAN image, and an arbitrary convex prior. For the unfolding stage, we introduce upsampling and downsampling layers that use geometric information encoded in the PAN image through residual networks. The backbone of our method is a multi-head attention residual network (MARNet), which replaces the proximity operator in the optimization scheme and combines multiple head attentions with residual learning to exploit image self-similarities via nonlocal operators defined in terms of patches. Additionally, we incorporate a post-processing module based on the MARNet architecture to further enhance the quality of the fused images. Experimental results on PRISMA, Quickbird, and WorldView2 datasets demonstrate the superior performance of our method and its ability to generalize across different sensor configurations and varying spatial and spectral resolutions. The source code will be available at https://github.com/TAMI-UIB/MARNet.