VIIS: Visible and Infrared Information Synthesis for Severe Low-light Image Enhancement

TL;DR

This paper introduces VIIS, a novel method using a diffusion model and dual-modal residual attention to enhance and fuse visible and infrared images in severe low-light conditions, outperforming existing techniques.

Contribution

The paper proposes a new task, VIIS, and a diffusion-based framework with a dual-modal residual attention mechanism for simultaneous information enhancement and fusion.

Findings

Outperforms state-of-the-art methods in qualitative and quantitative evaluations.

Effectively enhances and fuses visible and infrared images in low-light scenarios.

Demonstrates robustness across various low-light conditions.

Abstract

Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images, visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However, they primarily emphasize inter-modal complementation but neglect intra-modal enhancement, limiting the perceptual quality of output images. To address these limitations, we propose a novel task, dubbed visible and infrared information synthesis (VIIS), which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task, we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion. The code is available at https://github.com/Chenz418/VIIS.