CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion

TL;DR

CDDFuse introduces a novel multi-modality image fusion network that effectively decomposes and fuses cross-modality features using correlation-driven loss and dual-branch Transformer-CNN architecture, improving performance in various fusion tasks.

Contribution

The paper presents a new correlation-driven dual-branch Transformer-CNN network for multi-modality image fusion, effectively modeling cross-modality features and decomposing shared and specific features.

Findings

Achieves state-of-the-art results in infrared-visible and medical image fusion.

Enhances downstream tasks like semantic segmentation and object detection.

Demonstrates robustness across multiple fusion benchmarks.

Abstract

Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.com/Zhaozixiang1228/MMIF-CDDFuse.