TeSG: Textual Semantic Guidance for Infrared and Visible Image Fusion

TL;DR

TeSG introduces a novel text-guided image fusion framework that leverages large vision-language models to incorporate textual semantics at multiple levels, significantly improving the quality and utility of fused infrared and visible images for downstream tasks.

Contribution

The paper presents a new method, TeSG, which effectively integrates textual semantic information into infrared and visible image fusion using a multi-level guidance approach with novel modules.

Findings

TeSG outperforms existing methods in downstream detection and segmentation tasks.

The proposed modules improve the fusion quality by incorporating textual semantics.

TeSG demonstrates robustness and versatility across various datasets and scenarios.

Abstract

Infrared and visible image fusion (IVF) aims to combine complementary information from both image modalities, producing more informative and comprehensive outputs. Recently, text-guided IVF has shown great potential due to its flexibility and versatility. However, the effective integration and utilization of textual semantic information remains insufficiently studied. To tackle these challenges, we introduce textual semantics at two levels: the mask semantic level and the text semantic level, both derived from textual descriptions extracted by large Vision-Language Models (VLMs). Building on this, we propose Textual Semantic Guidance for infrared and visible image fusion, termed TeSG, which guides the image synthesis process in a way that is optimized for downstream tasks such as detection and segmentation. Specifically, TeSG consists of three core components: a Semantic Information Generator (SIG), a Mask-Guided Cross-Attention (MGCA) module, and a Text-Driven Attentional Fusion (TDAF) module. The SIG generates mask and text semantics based on textual descriptions. The MGCA module performs initial attention-based fusion of visual features from both infrared and visible images, guided by mask semantics. Finally, the TDAF module refines the fusion process with gated attention driven by text semantics. Extensive experiments demonstrate the competitiveness of our approach, particularly in terms of performance on downstream tasks, compared to existing state-of-the-art methods.