MdaIF: Robust One-Stop Multi-Degradation-Aware Image Fusion with Language-Driven Semantics

TL;DR

This paper introduces MdaIF, a novel image fusion framework that adaptively handles multiple weather-related degradations using a large language model and mixture-of-experts, significantly improving fusion robustness under adverse conditions.

Contribution

The paper presents a degradation-aware image fusion method leveraging a vision-language model and mixture-of-experts to adaptively address diverse weather degradations, a novel approach in multi-degradation scenarios.

Findings

Outperforms state-of-the-art fusion methods in complex weather conditions.

Effectively models multiple degradation scenarios with a mixture-of-experts.

Enhances fusion robustness using semantic priors from a vision-language model.

Abstract

Infrared and visible image fusion aims to integrate complementary multi-modal information into a single fused result. However, existing methods 1) fail to account for the degradation visible images under adverse weather conditions, thereby compromising fusion performance; and 2) rely on fixed network architectures, limiting their adaptability to diverse degradation scenarios. To address these issues, we propose a one-stop degradation-aware image fusion framework for multi-degradation scenarios driven by a large language model (MdaIF). Given the distinct scattering characteristics of different degradation scenarios (e.g., haze, rain, and snow) in atmospheric transmission, a mixture-of-experts (MoE) system is introduced to tackle image fusion across multiple degradation scenarios. To adaptively extract diverse weather-aware degradation knowledge and scene feature representations, collectively referred to as the semantic prior, we employ a pre-trained vision-language model (VLM) in our framework. Guided by the semantic prior, we propose degradation-aware channel attention module (DCAM), which employ degradation prototype decomposition to facilitate multi-modal feature interaction in channel domain. In addition, to achieve effective expert routing, the semantic prior and channel-domain modulated features are utilized to guide the MoE, enabling robust image fusion in complex degradation scenarios. Extensive experiments validate the effectiveness of our MdaIF, demonstrating superior performance over SOTA methods.