PoLAR: Polar-Decomposed Low-Rank Adapter Representation

TL;DR

PoLAR introduces a polar-decomposition inspired low-rank adaptation method that improves fine-tuning efficiency and performance of large-scale models across various benchmarks by better utilizing the adaptation subspace.

Contribution

The paper proposes PoLAR, a novel low-rank adaptation parameterization using polar decomposition and Riemannian optimization, enhancing convergence and performance in large-scale model fine-tuning.

Findings

PoLAR achieves faster convergence rates.

It improves performance on language understanding, reasoning, and math benchmarks.

Effective across models from 350M to 27B parameters.

Abstract

We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B.