Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks

TL;DR

This paper introduces a novel adaptive fusion framework that generates task-specific infrared-visible images guided by user instructions, improving multi-task performance without additional training.

Contribution

It proposes the T-OAR framework with the T-DPI module, enabling efficient, adaptable, and task-specific image fusion for multiple downstream tasks.

Findings

Enhanced performance in object detection, semantic segmentation, and salient object detection

Reduced computational costs for multi-task image fusion

Improved adaptability and task-specificity of fused images

Abstract

The primary value of infrared and visible image fusion technology lies in applying the fusion results to downstream tasks. However, existing methods face challenges such as increased training complexity and significantly compromised performance of individual tasks when addressing multiple downstream tasks simultaneously. To tackle this, we propose Task-Oriented Adaptive Regulation (T-OAR), an adaptive mechanism specifically designed for multi-task environments. Additionally, we introduce the Task-related Dynamic Prompt Injection (T-DPI) module, which generates task-specific dynamic prompts from user-input text instructions and integrates them into target representations. This guides the feature extraction module to produce representations that are more closely aligned with the specific requirements of downstream tasks. By incorporating the T-DPI module into the T-OAR framework, our approach generates fusion images tailored to task-specific requirements without the need for separate training or task-specific weights. This not only reduces computational costs but also enhances adaptability and performance across multiple tasks. Experimental results show that our method excels in object detection, semantic segmentation, and salient object detection, demonstrating its strong adaptability, flexibility, and task specificity. This provides an efficient solution for image fusion in multi-task environments, highlighting the technology's potential across diverse applications.