Align, Don't Divide: Revisiting the LoRA Architecture in Multi-Task Learning

TL;DR

This paper challenges the necessity of complex multi-adapter architectures in multi-task learning for LLMs, proposing a simpler aligned approach that improves performance by emphasizing shared representations.

Contribution

It introduces Align-LoRA, a novel method that aligns task representations within a shared adapter, demonstrating superior performance over complex multi-head or multi-adapter systems.

Findings

Simplified multi-head architecture outperforms complex systems.

Single-adapter LoRA with higher rank achieves competitive results.

Align-LoRA significantly surpasses baseline methods.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting Large Language Models (LLMs). In practice, LLMs are often required to handle a diverse set of tasks from multiple domains, a scenario naturally addressed by multi-task learning (MTL). Within this MTL context, a prevailing trend involves LoRA variants with multiple adapters or heads, which advocate for structural diversity to capture task-specific knowledge. Our findings present a direct challenge to this paradigm. We first show that a simplified multi-head architecture with high inter-head similarity substantially outperforms complex multi-adapter and multi-head systems. This leads us to question the multi-component paradigm itself, and we further demonstrate that a standard single-adapter LoRA, with a sufficiently increased rank, also achieves highly competitive performance. These results lead us to a new hypothesis: effective MTL generalization hinges on learning robust shared representations, not isolating task-specific features. To validate this, we propose Align-LoRA, which incorporates an explicit loss to align task representations within the shared adapter space. Experiments confirm that Align-LoRA significantly surpasses all baselines, establishing a simpler yet more effective paradigm for adapting LLMs to multiple tasks. The code is available at https://github.com/jinda-liu/Align-LoRA.