PanFormer: a Transformer Based Model for Pan-sharpening

TL;DR

This paper introduces PanFormer, a Transformer-based model for pan-sharpening that effectively fuses multi-spectral and panchromatic images, outperforming existing CNN-based methods in producing high-resolution multispectral images.

Contribution

The paper presents a novel Transformer-based architecture for pan-sharpening, leveraging self-attention and cross-attention mechanisms for improved feature extraction and fusion.

Findings

Outperforms CNN-based methods on GaoFen-2 and WorldView-3 datasets.

Demonstrates the effectiveness of Transformer models in image fusion tasks.

Achieves high-quality pan-sharpened images with detailed spectral and spatial information.

Abstract

Pan-sharpening aims at producing a high-resolution (HR) multi-spectral (MS) image from a low-resolution (LR) multi-spectral (MS) image and its corresponding panchromatic (PAN) image acquired by a same satellite. Inspired by a new fashion in recent deep learning community, we propose a novel Transformer based model for pan-sharpening. We explore the potential of Transformer in image feature extraction and fusion. Following the successful development of vision transformers, we design a two-stream network with the self-attention to extract the modality-specific features from the PAN and MS modalities and apply a cross-attention module to merge the spectral and spatial features. The pan-sharpened image is produced from the enhanced fused features. Extensive experiments on GaoFen-2 and WorldView-3 images demonstrate that our Transformer based model achieves impressive results and outperforms many existing CNN based methods, which shows the great potential of introducing Transformer to the pan-sharpening task. Codes are available at https://github.com/zhysora/PanFormer.