CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models

TL;DR

CE-LoRA is a novel fine-tuning method for large language models that reduces computational costs by approximating matrix operations and improving gradient accuracy, maintaining performance while enhancing efficiency.

Contribution

The paper introduces CE-LoRA, a computation-efficient fine-tuning algorithm that significantly lowers computational costs of LoRA without sacrificing accuracy.

Findings

CE-LoRA reduces training time compared to LoRA.

CE-LoRA maintains comparable model performance.

Theoretical convergence rate matches that of LoRA.

Abstract

Large Language Models (LLMs) demonstrate exceptional performance across various tasks but demand substantial computational resources even for fine-tuning computation. Although Low-Rank Adaptation (LoRA) significantly alleviates memory consumption during fine-tuning, its impact on computational cost reduction is limited. This paper identifies the computation of activation gradients as the primary bottleneck in LoRA's backward propagation and introduces the Computation-Efficient LoRA (CE-LoRA) algorithm, which enhances computational efficiency while preserving memory efficiency. CE-LoRA leverages two key techniques: Approximated Matrix Multiplication, which replaces dense multiplications of large and complete matrices with sparse multiplications involving only critical rows and columns, and the Double-LoRA technique, which reduces error propagation in activation gradients. Theoretically, CE-LoRA converges at the same rate as LoRA, $ \mathcal{O}(1/\sqrt{T}) $, where $T$ is the number of iteartions. Empirical evaluations confirm that CE-LoRA significantly reduces computational costs compared to LoRA without notable performance degradation.