MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning

TL;DR

This paper introduces MoSLD, a parameter-efficient multi-task learning model that uses shared LoRA matrices and dropout to improve performance and generalization across tasks and domains.

Contribution

MoSLD proposes a novel mixture-of-shared-LoRAs architecture with dropout, effectively addressing multi-task learning challenges while maintaining parameter efficiency.

Findings

Achieves superior performance in multi-task scenarios.

Demonstrates robust out-of-domain generalization.

Reduces parameter count compared to traditional MoE models.

Abstract

Recently, LoRA has emerged as a crucial technique for fine-tuning large pre-trained models, yet its performance in multi-task learning scenarios often falls short. In contrast, the MoE architecture presents a natural solution to this issue. However, it introduces challenges such as mutual interference of data across multiple domains and knowledge forgetting of various tasks. Additionally, MoE significantly increases the number of parameters, posing a computational cost challenge. Therefore, in this paper, we propose MoSLD, a mixture-of-shared-LoRAs model with a dropout strategy. MoSLD addresses these challenges by sharing the upper projection matrix in LoRA among different experts, encouraging the model to learn general knowledge across tasks, while still allowing the lower projection matrix to focus on the unique features of each task. The application of dropout alleviates the imbalanced update of parameter matrix and mitigates parameter overfitting in LoRA. Extensive experiments demonstrate that our model exhibits excellent performance in both single-task and multi-task scenarios, with robust out-of-domain generalization capabilities.