LoRA vs. Full Fine-Tuning: A Theoretical Perspective

TL;DR

This paper provides a theoretical analysis of LoRA, a parameter-efficient fine-tuning method, comparing its excess risk to full fine-tuning in linear regression, and explores conditions under which LoRA outperforms.

Contribution

It offers a theoretical framework for understanding LoRA's behavior, including regimes where it surpasses full fine-tuning and the impact of rank choice on generalization.

Findings

LoRA can achieve lower excess risk than full fine-tuning in certain regimes.

Low-rank differences between pretraining and downstream tasks benefit LoRA performance.

Small LoRA ranks can improve test accuracy despite limited expressivity.

Abstract

Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory and computation costs while often achieving performance close to full fine-tuning. Despite its widespread use, the theoretical behavior of LoRA is not yet well understood. In this paper, we study LoRA in a simple linear regression setting and compare its excess risk with that of full fine-tuning. Our analysis identifies regimes in which LoRA achieves lower excess risk than full fine-tuning in both overdetermined and underdetermined settings. Specifically, our theory predicts that LoRA can outperform full fine-tuning when the difference between the pretraining and the downstream tasks is effectively low-rank. We further show how the choice of LoRA rank affects generalization performance, explaining why using a very small rank can improve test accuracy in certain settings, even though it limits model expressivity. Finally, we support our theoretical results with experiments on practical tasks, suggesting that the identified tradeoffs and insights extend beyond linear regression.