CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks

TL;DR

CoLA introduces a dual-path PEFT framework that enhances multimodal foundation model adaptation by capturing cross-modal interactions efficiently, outperforming previous methods across vision-language and audio-visual benchmarks.

Contribution

The paper presents CoLA, a novel PEFT method with inter-modal and intra-modal pathways, enabling effective and parameter-efficient multimodal model adaptation.

Findings

CoLA outperforms LoRA by around 3% on vision-language benchmarks.

CoLA achieves a 2% improvement on audio-visual tasks.

First multi-task PEFT framework for visual grounding.

Abstract

Foundation models have revolutionized AI, but adapting them efficiently for multimodal tasks, particularly in dual-stream architectures composed of unimodal encoders, such as DINO and BERT, remains a significant challenge. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) enable lightweight adaptation, yet they operate in isolation within each modality, limiting their ability in capturing cross-modal interactions. In this paper, we take a step in bridging this gap with Cross-Modal Low-Rank Adaptation (CoLA), a novel PEFT framework that extends LoRA by introducing a dedicated inter-modal adaptation pathway alongside the standard intra-modal one. This dual-path design enables CoLA to adapt unimodal foundation models to multimodal tasks effectively, without interference between modality-specific and cross-modal learning. We evaluate CoLA across a range of vision-language (RefCOCO, RefCOCO+, RefCOCOg) and audio-visual (AVE, AVS) benchmarks, where it consistently outperforms LORA, achieving a relative gain of around 3\% and 2\%, respectively, while maintaining parameter efficiency. Notably, CoLA enables the first multi-task PEFT framework for visual grounding, bridging a key gap in efficient multimodal adaptation.