IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks

TL;DR

IV-tuning is a parameter-efficient method that leverages pre-trained visual models to improve infrared-visible tasks, maintaining feature diversity and outperforming existing methods with minimal trainable parameters.

Contribution

The paper introduces IV-tuning, a novel approach that freezes most pre-trained model parameters, enabling effective IR-VIS task learning with only 3% trainable parameters and enhanced generalization.

Findings

Outperforms previous state-of-the-art methods.

Achieves superior generalization and scalability.

Uses only 3% trainable backbone parameters.

Abstract

Existing infrared and visible (IR-VIS) methods inherit the general representations of Pre-trained Visual Models (PVMs) to facilitate complementary learning. However, our analysis indicates that under the full fine-tuning paradigm, the feature space becomes highly constrained and low-ranked, which has been proven to seriously impair generalization. One remedy is to freeze the parameters, which preserves pretrained knowledge and helps maintain feature diversity. To this end, we propose IV-tuning, to parameter-efficiently harness PVMs for various IR-VIS downstream tasks, including salient object detection, semantic segmentation, and object detection. Extensive experiments across various settings demonstrate that IV-tuning outperforms previous state-of-the-art methods, and exhibits superior generalization and scalability. Remarkably, with only a single backbone, IV-tuning effectively facilitates the complementary learning of infrared and visible modalities with merely 3% trainable backbone parameters, and achieves superior computational efficiency compared to conventional IR-VIS paradigms.