On the Convergence Rate of LoRA Gradient Descent

TL;DR

This paper provides the first non-asymptotic convergence analysis of the original LoRA gradient descent algorithm, showing it converges to a stationary point at a rate of O(1/log T) without assuming Lipschitz smoothness.

Contribution

It introduces a novel non-asymptotic convergence analysis for LoRA gradient descent, removing previous restrictive assumptions and establishing a convergence rate.

Findings

LoRA gradient descent converges at rate O(1/log T)

The analysis applies to the original LoRA algorithm used in practice

Numerical experiments support the theoretical convergence rate

Abstract

The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the \textit{original LoRA gradient descent} algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the ``Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate $O(\frac{1}{\log T})$, where $T$ is the number of iterations. We conduct numerical experiments to validate our theoretical findings.