IVGF: The Fusion-Guided Infrared and Visible General Framework

TL;DR

The paper introduces IVGF, a versatile fusion framework for infrared and visible images that enhances high-level vision tasks like segmentation and detection by leveraging foundation models and novel modules for feature and token enhancement, along with an attention-guided fusion strategy.

Contribution

It presents a general, extendable framework that improves dual-modality vision tasks through innovative feature, token, and fusion modules, outperforming existing methods.

Findings

Outperforms state-of-the-art dual-modality methods in segmentation and detection.

Effective modules validated through extensive ablation studies.

Enhanced anti-missing modality capability in dual-modality segmentation.

Abstract

Infrared and visible dual-modality tasks such as semantic segmentation and object detection can achieve robust performance even in extreme scenes by fusing complementary information. Most current methods design task-specific frameworks, which are limited in generalization across multiple tasks. In this paper, we propose a fusion-guided infrared and visible general framework, IVGF, which can be easily extended to many high-level vision tasks. Firstly, we adopt the SOTA infrared and visible foundation models to extract the general representations. Then, to enrich the semantics information of these general representations for high-level vision tasks, we design the feature enhancement module and token enhancement module for feature maps and tokens, respectively. Besides, the attention-guided fusion module is proposed for effectively fusing by exploring the complementary information of two modalities. Moreover, we also adopt the cutout&mix augmentation strategy to conduct the data augmentation, which further improves the ability of the model to mine the regional complementary between the two modalities. Extensive experiments show that the IVGF outperforms state-of-the-art dual-modality methods in the semantic segmentation and object detection tasks. The detailed ablation studies demonstrate the effectiveness of each module, and another experiment explores the anti-missing modality ability of the proposed method in the dual-modality semantic segmentation task.