Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC

TL;DR

This paper introduces LoRSum, a memory-efficient method for low-rank model adaptation that improves upon LoRA by using proximal sub-problems and structured preconditioning, achieving better performance with less memory.

Contribution

We propose LoRSum, a novel proximal subroutine for low-rank adaptation that unifies several preconditioning methods and enhances LoRA with memory-efficient structured metrics.

Findings

LoRSum matches or outperforms LoRA on various tasks.

Structured metrics like K-FAC improve efficiency while maintaining performance.

The method reduces memory usage compared to full SVD projections.

Abstract

Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning. We propose LoRSum, a memory-efficient subroutine that closes this gap for gradient descent by casting LoRA optimization as a proximal sub-problem and solving it efficiently with alternating least squares updates, which we prove to be an implicit block power method. We recover several recently proposed preconditioning methods for LoRA as special cases, and show that LoRSum can also be used for updating a low-rank momentum. In order to address full steps with preconditioned gradient descent, we propose a scaled variant of LoRSum that uses structured metrics such as K-FAC and Shampoo, and we show that storing the diagonal of these metrics still allows them to perform well while remaining memory-efficient. Experiments on a synthetic task, CIFAR-100, and language-model fine-tuning on GLUE, SQuAD v2, and WikiText-103, show that our method can match or improve LoRA baselines given modest compute overhead, while avoiding full-matrix SVD projections and retaining LoRA-style parameter efficiency.