MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation

TL;DR

MLorc introduces a memory-efficient training method for large language models by compressing momentum, enabling full-parameter learning with reduced memory use and improved performance over existing methods.

Contribution

The paper proposes MLorc, a novel momentum low-rank compression technique that preserves training dynamics and enhances memory efficiency in LLM fine-tuning.

Findings

MLorc outperforms other memory-efficient methods in experiments.

It matches or exceeds full fine-tuning performance at small ranks.

MLorc generalizes well across different optimizers.

Abstract

With increasing size of large language models (LLMs), full-parameter fine-tuning imposes substantial memory demands. To alleviate this, we propose a novel memory-efficient training paradigm called Momentum Low-rank compression (MLorc). The key idea of MLorc is to compress and reconstruct the momentum of matrix parameters during training to reduce memory consumption. Compared to LoRA, MLorc avoids enforcing a fixed-rank constraint on weight update matrices and thus enables full-parameter learning. Compared to GaLore, MLorc directly compress the momentum rather than gradients, thereby better preserving the training dynamics of full-parameter fine-tuning. We provide a theoretical guarantee for its convergence under mild assumptions. Empirically, MLorc consistently outperforms other memory-efficient training methods, matches or even exceeds the performance of full fine-tuning at small ranks (e.g., $r=4$), and generalizes well across different optimizers, all while not compromising time or memory efficiency.