UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning

TL;DR

UniRGB-IR introduces a scalable adapter-based framework that enhances pre-trained RGB foundation models for diverse RGB-IR semantic tasks, achieving state-of-the-art results with improved generalization.

Contribution

The paper proposes a novel adapter mechanism with MFP and SFI modules to effectively incorporate multi-modal features into frozen pre-trained models for RGB-IR tasks.

Findings

Achieves state-of-the-art performance on various RGB-IR semantic tasks.

Effectively incorporates multi-scale features via adapter modules.

Maintains high scalability and generalization across tasks.

Abstract

Semantic analysis on visible (RGB) and infrared (IR) images has gained significant attention due to their enhanced accuracy and robustness under challenging conditions including low-illumination and adverse weather. However, due to the lack of pre-trained foundation models on the large-scale infrared image datasets, existing methods prefer to design task-specific frameworks and directly fine-tune them with pre-trained foundation models on their RGB-IR semantic relevance datasets, which results in poor scalability and limited generalization. To address these limitations, we propose UniRGB-IR, a scalable and efficient framework for RGB-IR semantic tasks that introduces a novel adapter mechanism to effectively incorporate rich multi-modal features into pre-trained RGB-based foundation models. Our framework comprises three key components: a vision transformer (ViT) foundation model, a Multi-modal Feature Pool (MFP) module, and a Supplementary Feature Injector (SFI) module. The MFP and SFI modules cooperate with each other as an adpater to effectively complement the ViT features with the contextual multi-scale features. During training process, we freeze the entire foundation model to inherit prior knowledge and only optimize the MFP and SFI modules. Furthermore, to verify the effectiveness of our framework, we utilize the ViT-Base as the pre-trained foundation model to perform extensive experiments. Experimental results on various RGB-IR semantic tasks demonstrate that our method can achieve state-of-the-art performance.