RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts

TL;DR

This paper introduces RepLoRA, a reparameterization technique for LoRA inspired by Mixture of Experts, which accelerates training and improves performance with limited data in fine-tuning large models.

Contribution

We provide a theoretical connection between LoRA and Mixture of Experts, and propose RepLoRA, a novel reparameterization method that enhances low-rank adaptation efficiency and effectiveness.

Findings

RepLoRA outperforms vanilla LoRA across multiple domains.

RepLoRA achieves similar performance to LoRA with only 30% of training data.

RepLoRA reduces data requirements from exponential to polynomial scale.

Abstract

Low-rank Adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remained limited. This paper presents a theoretical analysis of LoRA by examining its connection to the Mixture of Experts models. Under this framework, we show that simple reparameterizations of the LoRA matrices can notably accelerate the low-rank matrix estimation process. In particular, we prove that reparameterization can reduce the data needed to achieve a desired estimation error from an exponential to a polynomial scale. Motivated by this insight, we propose Reparameterized Low-Rank Adaptation (RepLoRA), which incorporates lightweight MLPs to reparameterize the LoRA matrices. Extensive experiments across multiple domains demonstrate that RepLoRA consistently outperforms vanilla LoRA. Notably, with limited data, RepLoRA surpasses LoRA by a margin of up to 40.0% and achieves LoRA's performance with only 30.0% of the training data, highlighting both the theoretical and empirical robustness of our PEFT method.