M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension

TL;DR

M2IST introduces a parameter-efficient method for referring expression comprehension that enhances multi-modal interaction while significantly reducing computational costs compared to full fine-tuning.

Contribution

The paper proposes M2IST, a novel multi-modal side-tuning approach with M3ISAs, enabling efficient vision-language alignment without extensive parameter updates.

Findings

Outperforms full fine-tuning and other PETL methods in efficiency and performance.

Uses only 2.11% of tunable parameters, reducing GPU memory and training time.

Maintains competitive accuracy in referring expression comprehension.

Abstract

Referring expression comprehension (REC) is a vision-language task to locate a target object in an image based on a language expression. Fully fine-tuning general-purpose pre-trained vision-language foundation models for REC yields impressive performance but becomes increasingly costly. Parameter-efficient transfer learning (PETL) methods have shown strong performance with fewer tunable parameters. However, directly applying PETL to REC faces two challenges: (1) insufficient multi-modal interaction between pre-trained vision-language foundation models, and (2) high GPU memory usage due to gradients passing through the heavy vision-language foundation models. To this end, we present M2IST: Multi-Modal Interactive Side-Tuning with M3ISAs: Mixture of Multi-Modal Interactive Side-Adapters. During fine-tuning, we fix the pre-trained uni-modal encoders and update M3ISAs to enable efficient vision-language alignment for REC. Empirical results reveal that M2IST achieves better performance-efficiency trade-off than full fine-tuning and other PETL methods, requiring only 2.11\% tunable parameters, 39.61\% GPU memory, and 63.46\% training time while maintaining competitive performance. Our code is released at https://github.com/xuyang-liu16/M2IST.