AsymLoRA: Harmonizing Data Conflicts and Commonalities in MLLMs

TL;DR

AsymLoRA is a novel parameter-efficient fine-tuning method for MLLMs that effectively manages data conflicts and commonalities, improving performance across diverse multimodal tasks.

Contribution

It introduces asymmetric LoRA with task-specific and shared projections to unify handling of conflicting and common dataset features in MLLMs.

Findings

Outperforms vanilla LoRA and LoRA-MoE in accuracy

Enhances model efficiency and adaptability

Achieves superior results on multiple benchmarks

Abstract

Effective instruction fine-tuning on diverse image-text datasets is crucial for developing a versatile Multimodal Large Language Model (MLLM), where dataset composition dictates the model's adaptability across multimodal tasks. However, complex datasets often contain inherent conflicts -- stemming from modality-specific optimization objectives -- and latent commonalities that enable cross-task transfer, which most existing approaches handle separately. To bridge this gap, we introduce AsymLoRA, a parameter-efficient tuning framework that unifies knowledge modularization and cross-modal coordination via asymmetric LoRA: task-specific low-rank projections (matrix B) that preserve distinct adaptation pathways for conflicting objectives, and a shared projection (matrix A) that consolidates cross-modal commonalities. Extensive evaluations demonstrate that AsymLoRA consistently surpasses both vanilla LoRA, which captures only commonalities, and LoRA-MoE, which focuses solely on conflicts, achieving superior model performance and system efficiency across diverse benchmarks.\href{Code}{https://github.com/Clin0212/HydraLoRA/blob/main/MLLM-HydraLoRA/README.md}.