MokA: Multimodal Low-Rank Adaptation for MLLMs

TL;DR

MokA introduces a multimodal-aware fine-tuning strategy for MLLMs that enhances both unimodal and cross-modal adaptation, leading to improved efficiency and performance across various multimodal scenarios and backbones.

Contribution

The paper proposes MokA, a novel low-rank adaptation method that considers multimodal characteristics, addressing limitations of previous LLM-based approaches.

Findings

Consistent performance improvements across three multimodal scenarios.

Effective enhancement of cross-modal interactions.

Versatility demonstrated on multiple LLM backbones.

Abstract

In this paper, we reveal that most current efficient multimodal fine-tuning methods are hindered by a key limitation: they are directly borrowed from LLMs, often neglecting the intrinsic differences of multimodal scenarios and even affecting the full utilization of all modalities. Inspired by our empirical observation, we argue that unimodal adaptation and cross-modal adaptation are two essential parts for the effective fine-tuning of MLLMs. From this perspective, we propose Multimodal low-rank Adaptation (MokA), a multimodal-aware efficient fine-tuning strategy that takes multimodal characteristics into consideration. It compresses unimodal information by modality-specific parameters while explicitly enhancing cross-modal interaction, ensuring both unimodal and cross-modal adaptation. Extensive experiments cover three representative multimodal scenarios (audio-visual-text, visual-text, and speech-text), and multiple LLM backbones (LLaMA2/3, Qwen2, Qwen2.5-VL, etc). Consistent improvements indicate the efficacy and versatility of the proposed method. Ablation studies and efficiency evaluation are also conducted to fully asses our method. Overall, we think MokA provides a more targeted solution for efficient adaptation of MLLMs, paving the way for further exploration. The project page is at https://gewu-lab.github.io/MokA.