All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark

TL;DR

This paper introduces a unified all-weather multi-modality image fusion framework that effectively handles diverse weather conditions and provides a large-scale benchmark dataset for robust scene understanding.

Contribution

It proposes an end-to-end model that decomposes images into components, uses physically-aware modules for feature enhancement, and introduces a 100,000 image dataset for comprehensive evaluation.

Findings

Outperforms existing methods in image fusion quality.

Enhances downstream tasks like object detection and segmentation.

Demonstrates robustness across various weather conditions.

Abstract

Multi-modality image fusion (MMIF) combines complementary information from different image modalities to provide a comprehensive and objective interpretation of scenes. However, existing fusion methods cannot resist different weather interferences in real-world scenes, limiting their practical applicability. To bridge this gap, we propose an end-to-end, unified all-weather MMIF model. Rather than focusing solely on pixel-level recovery, our method emphasizes maximizing the representation of key scene information through joint feature fusion and restoration. Specifically, we first decompose images into low-rank and sparse components, enabling effective feature separation for enhanced multi-modality perception. During feature recovery, we introduce a physically-aware clear feature prediction module, inferring variations in light transmission via illumination and reflectance. Clear features generated by the network are used to enhance salient information representation. We also construct a large-scale MMIF dataset with 100,000 image pairs comprehensively across rain, haze, and snow conditions, as well as covering various degradation levels and diverse scenes. Experimental results in both real-world and synthetic scenes demonstrate that the proposed method excels in image fusion and downstream tasks such as object detection, semantic segmentation, and depth estimation. The code is available at https://github.com/ixilai/AWFusion.