PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion

TL;DR

This paper introduces the Patch Pyramid Transformer (PPT), a novel architecture that combines local patch-based and global pyramid features to improve image fusion performance, outperforming existing methods without needing retraining for different tasks.

Contribution

The paper proposes a new Patch Pyramid Transformer that effectively captures local and non-local image features for versatile image fusion applications, with superior experimental results.

Findings

Achieves state-of-the-art results on multiple image fusion benchmarks.

Demonstrates robustness across different fusion tasks without retraining.

Outperforms existing fusion methods in key evaluation metrics.

Abstract

The Transformer architecture has witnessed a rapid development in recent years, outperforming the CNN architectures in many computer vision tasks, as exemplified by the Vision Transformers (ViT) for image classification. However, existing visual transformer models aim to extract semantic information for high-level tasks, such as classification and detection.These methods ignore the importance of the spatial resolution of the input image, thus sacrificing the local correlation information of neighboring pixels. In this paper, we propose a Patch Pyramid Transformer(PPT) to effectively address the above issues.Specifically, we first design a Patch Transformer to transform the image into a sequence of patches, where transformer encoding is performed for each patch to extract local representations. In addition, we construct a Pyramid Transformer to effectively extract the non-local information from the entire image. After obtaining a set of multi-scale, multi-dimensional, and multi-angle features of the original image, we design the image reconstruction network to ensure that the features can be reconstructed into the original input. To validate the effectiveness, we apply the proposed Patch Pyramid Transformer to image fusion tasks. The experimental results demonstrate its superior performance, compared to the state-of-the-art fusion approaches, achieving the best results on several evaluation indicators. Thanks to the underlying representational capacity of the PPT network, it can directly be applied to different image fusion tasks without redesigning or retraining the network.