MagicFuse: Single Image Fusion for Visual and Semantic Reinforcement

TL;DR

MagicFuse introduces a novel single-image fusion framework that enhances visual and semantic scene understanding from a single low-quality visible image using diffusion models and knowledge reinforcement.

Contribution

It extends traditional multi-modal fusion to a knowledge-based approach capable of deriving cross-spectral scene representations from a single image.

Findings

Achieves comparable or better performance than multi-modal methods.

Effectively mines scene information obscured in visible spectrum.

Supports downstream semantic decision-making.

Abstract

This paper focuses on a highly practical scenario: how to continue benefiting from the advantages of multi-modal image fusion under harsh conditions when only visible imaging sensors are available. To achieve this goal, we propose a novel concept of single-image fusion, which extends conventional data-level fusion to the knowledge level. Specifically, we develop MagicFuse, a novel single image fusion framework capable of deriving a comprehensive cross-spectral scene representation from a single low-quality visible image. MagicFuse first introduces an intra-spectral knowledge reinforcement branch and a cross-spectral knowledge generation branch based on the diffusion models. They mine scene information obscured in the visible spectrum and learn thermal radiation distribution patterns transferred to the infrared spectrum, respectively. Building on them, we design a multi-domain knowledge fusion branch that integrates the probabilistic noise from the diffusion streams of these two branches, from which a cross-spectral scene representation can be obtained through successive sampling. Then, we impose both visual and semantic constraints to ensure that this scene representation can satisfy human observation while supporting downstream semantic decision-making. Extensive experiments show that our MagicFuse achieves visual and semantic representation performance comparable to or even better than state-of-the-art fusion methods with multi-modal inputs, despite relying solely on a single degraded visible image. The code is publicly available at https://github.com/zhayanping/MagicFuse.