Rethinking the Evaluation of Visible and Infrared Image Fusion

TL;DR

This paper introduces a segmentation-oriented evaluation method for visible and infrared image fusion that leverages semantic segmentation to better assess fusion quality, revealing that many current methods do not outperform using visible images alone.

Contribution

The paper proposes a novel evaluation approach using semantic segmentation labels to assess VIF methods, providing more practical insights into their real-world performance.

Findings

Many VIF methods perform comparably or worse than visible images alone.

The segmentation-oriented evaluation reveals limitations of current VIF methods.

Metrics like $Q_{ABF}$ and $Q_{VIFF}$ correlate well with segmentation-based assessments.

Abstract

Visible and Infrared Image Fusion (VIF) has garnered significant interest across a wide range of high-level vision tasks, such as object detection and semantic segmentation. However, the evaluation of VIF methods remains challenging due to the absence of ground truth. This paper proposes a Segmentation-oriented Evaluation Approach (SEA) to assess VIF methods by incorporating the semantic segmentation task and leveraging segmentation labels available in latest VIF datasets. Specifically, SEA utilizes universal segmentation models, capable of handling diverse images and classes, to predict segmentation outputs from fused images and compare these outputs with segmentation labels. Our evaluation of recent VIF methods using SEA reveals that their performance is comparable or even inferior to using visible images only, despite nearly half of the infrared images demonstrating better performance than visible images. Further analysis indicates that the two metrics most correlated to our SEA are the gradient-based fusion metric $Q_{\text{ABF}}$ and the visual information fidelity metric $Q_{\text{VIFF}}$ in conventional VIF evaluation metrics, which can serve as proxies when segmentation labels are unavailable. We hope that our evaluation will guide the development of novel and practical VIF methods. The code has been released in \url{https://github.com/Yixuan-2002/SEA/}.