LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning

TL;DR

LoFT introduces a low-rank adaptation method that mimics full fine-tuning by aligning optimizer dynamics, achieving better accuracy without extra hyperparameters or increased inference costs.

Contribution

LoFT is a novel low-rank adaptation technique that aligns optimizer moments with full fine-tuning, eliminating hyperparameter tuning and improving performance.

Findings

Significantly narrows the performance gap with full fine-tuning.

Outperforms standard LoRA methods consistently.

Maintains the same inference cost as LoRA.

Abstract

Large pre-trained models are commonly adapted to downstream tasks using parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable low-rank matrices instead of updating all weights. While LoRA dramatically reduces trainable parameters with little overhead, it can still underperform full fine-tuning in accuracy and often converges more slowly. We introduce LoFT, a novel low-rank adaptation method that behaves like full fine-tuning by aligning the optimizer's internal dynamics with those of updating all model weights. LoFT not only learns weight updates in a low-rank subspace (like LoRA) but also properly projects the optimizer's first and second moments (Adam's momentum and variance) into the same subspace, mirroring full-model updates. By aligning the low-rank update itself with the full update, LoFT eliminates the need for tuning extra hyperparameters, e.g., the LoRA scaling factor $\alpha$. Empirically, this approach substantially narrows the performance gap between adapter-based tuning and full fine-tuning and consistently outperforms standard LoRA-style methods, all without increasing inference cost.