Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection

TL;DR

This paper introduces IA-VFDnet, a hybrid CNN-Transformer framework that enables high-quality multimodal image fusion detection without the need for image registration, improving wildfire detection performance.

Contribution

It proposes a novel registration-free fusion framework with a unified feature matching and fusion module, advancing multimodal detection capabilities.

Findings

Achieves superior detection performance on the M3FD dataset.

First to establish an unregistered multimodal smoke and wildfire detection benchmark.

Demonstrates effectiveness without costly image registration processes.

Abstract

Multimodal fusion detection always places high demands on the imaging system and image pre-processing, while either a high-quality pre-registration system or image registration processing is costly. Unfortunately, the existing fusion methods are designed for registered source images, and the fusion of inhomogeneous features, which denotes a pair of features at the same spatial location that expresses different semantic information, cannot achieve satisfactory performance via these methods. As a result, we propose IA-VFDnet, a CNN-Transformer hybrid learning framework with a unified high-quality multimodal feature matching module (AKM) and a fusion module (WDAF), in which AKM and DWDAF work in synergy to perform high-quality infrared-aware visible fusion detection, which can be applied to smoke and wildfire detection. Furthermore, experiments on the M3FD dataset validate the superiority of the proposed method, with IA-VFDnet achieving the best detection performance than other state-of-the-art methods under conventional registered conditions. In addition, the first unregistered multimodal smoke and wildfire detection benchmark is openly available in this letter.