Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters

TL;DR

This paper introduces Mixture of Low-rank Adapters (MoLA), a novel method to effectively train unified models on heterogeneous data by mitigating conflicts through task-specific adapters and a task-wise decorrelation loss.

Contribution

It proposes two variants, MoLA-Grad and MoLA-Router, to handle target-aware and target-agnostic scenarios, advancing multi-task learning with low-rank adapters.

Findings

MoLA outperforms previous state-of-the-art methods.

MoLA effectively mitigates training conflicts among heterogeneous data.

In-depth analysis reveals the working mechanism of MoLA.

Abstract

Training a unified model to take multiple targets into account is a trend towards artificial general intelligence. However, how to efficiently mitigate the training conflicts among heterogeneous data collected from different domains or tasks remains under-explored. In this study, we explore to leverage Mixture of Low-rank Adapters (MoLA) to mitigate conflicts in heterogeneous data training, which requires to jointly train the multiple low-rank adapters and their shared backbone. Specifically, we introduce two variants of MoLA, namely, MoLA-Grad and MoLA-Router, to respectively handle the target-aware and target-agnostic scenarios during inference. The former uses task identifiers to assign personalized low-rank adapters to each task, disentangling task-specific knowledge towards their adapters, thereby mitigating heterogeneity conflicts. The latter uses a novel Task-wise Decorrelation (TwD) loss to intervene the router to learn oriented weight combinations of adapters to homogeneous tasks, achieving similar effects. We conduct comprehensive experiments to verify the superiority of MoLA over previous state-of-the-art methods and present in-depth analysis on its working mechanism. Source code is available at: https://github.com/MediaBrain-SJTU/MoLA