Rank-Accuracy Trade-off for LoRA: A Gradient-Flow Analysis

TL;DR

This paper provides a theoretical analysis of how the rank of LoRA updates influences accuracy, using gradient flow dynamics to establish explicit relationships and deepen understanding of low-rank fine-tuning methods.

Contribution

It offers the first rigorous dynamical systems analysis of LoRA, deriving explicit formulas linking update rank to accuracy for specific loss functions.

Findings

Derived closed-form relationships between rank and accuracy.

Established that gradient flow equations are identical for simultaneous and sequential updates.

Provided theoretical insights into low-rank approximation effects on fine-tuning accuracy.

Abstract

Previous empirical studies have shown that LoRA achieves accuracy comparable to full-parameter methods on downstream fine-tuning tasks, even for rank-1 updates. By contrast, the theoretical underpinnings of the dependence of LoRA's accuracy on update rank remain relatively unexplored. In this work, we compare the accuracy of rank-r LoRA updates against full-parameter updates for fine-tuning tasks from a dynamical systems perspective. We perform gradient flow analysis in both full-rank and low-rank regimes to establish explicit relationships between rank and accuracy for two loss functions under LoRA. While gradient flow equations for LoRA are presented in prior work, we rigorously derive their form and show that they are identical for simultaneous and sequential LoRA parameter updates. We then use the resulting dynamical system equations to obtain closed-form relationships between LoRA rank and accuracy for trace-squared and Frobenius-norm low-rank approximation loss functions.