LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

TL;DR

This paper introduces LoRA-RITE, an adaptive matrix preconditioning method for LoRA optimization that achieves transformation invariance, leading to more efficient learning and improved performance across various large language models.

Contribution

LoRA-RITE is a novel transformation-invariant optimization method for LoRA that enhances fine-tuning efficiency and effectiveness in large language models.

Findings

LoRA-RITE improves accuracy by up to 4.6% on Super-Natural Instructions.

It achieves consistent performance gains across multiple LLM benchmarks.

Theoretical analysis confirms the benefits of transformation invariance in LoRA optimization.

Abstract

Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6\% accuracy gain on Super-Natural Instructions and 3.5\% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA).