Self-supervised Multiplex Consensus Mamba for General Image Fusion

TL;DR

The paper introduces SMC-Mamba, a self-supervised framework for general image fusion that enhances feature preservation and integration across modalities, improving performance in various fusion tasks without added complexity.

Contribution

It proposes a novel self-supervised multiplex consensus framework with modules for feature enhancement, dynamic expert collaboration, and a new contrastive loss for effective multi-modal image fusion.

Findings

Outperforms state-of-the-art in multiple image fusion tasks

Enhances downstream task performance such as object detection

Preserves high-frequency details effectively

Abstract

Image fusion integrates complementary information from different modalities to generate high-quality fused images, thereby enhancing downstream tasks such as object detection and semantic segmentation. Unlike task-specific techniques that primarily focus on consolidating inter-modal information, general image fusion needs to address a wide range of tasks while improving performance without increasing complexity. To achieve this, we propose SMC-Mamba, a Self-supervised Multiplex Consensus Mamba framework for general image fusion. Specifically, the Modality-Agnostic Feature Enhancement (MAFE) module preserves fine details through adaptive gating and enhances global representations via spatial-channel and frequency-rotational scanning. The Multiplex Consensus Cross-modal Mamba (MCCM) module enables dynamic collaboration among experts, reaching a consensus to efficiently integrate complementary information from multiple modalities. The cross-modal scanning within MCCM further strengthens feature interactions across modalities, facilitating seamless integration of critical information from both sources. Additionally, we introduce a Bi-level Self-supervised Contrastive Learning Loss (BSCL), which preserves high-frequency information without increasing computational overhead while simultaneously boosting performance in downstream tasks. Extensive experiments demonstrate that our approach outperforms state-of-the-art (SOTA) image fusion algorithms in tasks such as infrared-visible, medical, multi-focus, and multi-exposure fusion, as well as downstream visual tasks.