A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models

TL;DR

This paper introduces a novel training strategy for Mixture-of-Low-Rank-Adapters (MoE-LoRA) that enhances robustness and feature learning in fine-tuning foundation models, leveraging Riemannian preconditioners for stability.

Contribution

It proposes a Riemannian preconditioning-based training method for MoE-LoRA, improving robustness and effectiveness in fine-tuning large models.

Findings

Enhanced stability during training and inference.

Improved performance across downstream tasks.

Effective with SGD and AdamW optimizers.

Abstract

In order to streamline the fine-tuning of foundation models, Low-Rank Adapters (LoRAs) have been substantially adopted across various fields, including instruction tuning and domain adaptation. The underlying concept of LoRA involves decomposing a full-rank matrix into the product of two lower-rank matrices, which reduces storage consumption and accelerates the training process. Furthermore, to address the limited expressive capacity of LoRA, the Mixture-of-Expert (MoE) has been introduced for incorporating multiple LoRA adapters. The integration of LoRA experts leads to a visible improvement across several downstream scenes. However, the mixture of LoRAs (MoE-LoRA) still exhibits its low robustness during tuning and inferring. Inspired by the Riemannian Preconditioners which train LoRA as a sub-space projector, we propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning procedure by multi-space projections. Examinations on SGD and AdamW optimizers demonstrate the effectiveness of our methodology. Source code is available at https://github.com/THUDM/MoELoRA_Riemannian.