Design and Evaluation of Deep Learning-Based Dual-Spectrum Image Fusion Methods

TL;DR

This paper introduces a new dual-spectrum dataset and a comprehensive evaluation framework for deep learning-based image fusion, emphasizing downstream task performance and fair comparison of algorithms in challenging scenes.

Contribution

It provides a high-quality dual-spectrum dataset, a task-aware evaluation framework, and a thorough analysis of fusion methods across diverse scenarios.

Findings

Fusion models optimized for downstream tasks outperform general metric-based models.

Some algorithms excel in traditional metrics but underperform in downstream tasks.

The proposed framework reveals limitations of current evaluation practices.

Abstract

Visible images offer rich texture details, while infrared images emphasize salient targets. Fusing these complementary modalities enhances scene understanding, particularly for advanced vision tasks under challenging conditions. Recently, deep learning-based fusion methods have gained attention, but current evaluations primarily rely on general-purpose metrics without standardized benchmarks or downstream task performance. Additionally, the lack of well-developed dual-spectrum datasets and fair algorithm comparisons hinders progress. To address these gaps, we construct a high-quality dual-spectrum dataset captured in campus environments, comprising 1,369 well-aligned visible-infrared image pairs across four representative scenarios: daytime, nighttime, smoke occlusion, and underpasses. We also propose a comprehensive and fair evaluation framework that integrates fusion speed, general metrics, and object detection performance using the lang-segment-anything model to ensure fairness in downstream evaluation. Extensive experiments benchmark several state-of-the-art fusion algorithms under this framework. Results demonstrate that fusion models optimized for downstream tasks achieve superior performance in target detection, especially in low-light and occluded scenes. Notably, some algorithms that perform well on general metrics do not translate to strong downstream performance, highlighting limitations of current evaluation practices and validating the necessity of our proposed framework. The main contributions of this work are: (1)a campus-oriented dual-spectrum dataset with diverse and challenging scenes; (2) a task-aware, comprehensive evaluation framework; and (3) thorough comparative analysis of leading fusion methods across multiple datasets, offering insights for future development.