Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training

TL;DR

The paper introduces Q3R, a novel low-rank regularizer inspired by IRLS, enabling effective low-rank training of deep models with minimal performance loss and computational overhead.

Contribution

Q3R is a new quadratic regularizer that induces low-rank structure in model weights, allowing prescribed low-rank training while maintaining competitive accuracy.

Findings

Q3R enables training to specific low ranks with minor accuracy drops.

Q3R achieves comparable performance to dense models with reduced parameters.

Effective on vision and language transformer models.

Abstract

Parameter-efficient training based on low-rank optimization has become a highly successful tool for fine-tuning large deep learning models. However, these methods often fail for low-rank pre-training, where simultaneously maintaining low-rank weight structure and optimizing the task objective remains challenging. We propose the $\textit{Quadratic Reweighted Rank Regularizer}$ ($\texttt{Q3R}$), which leads to a novel low-rank-inducing training strategy inspired by the Iteratively Reweighted Least Squares (IRLS) framework. $\texttt{Q3R}$ is based on a quadratic regularizer term that majorizes a smoothed log-determinant rank surrogate. Unlike other low-rank training techniques, $\texttt{Q3R}$ can train weight matrices to prescribed low target ranks while achieving predictive performance comparable to dense models, with small computational overhead and full compatibility with existing architectures. For example, we demonstrate a $\texttt{Q3R}$-regularized ViT-Tiny experiment where truncating the model to $60\%$ and $80\%$ of its parameters results in only minor absolute accuracy drops of $1.3\%$ and $4\%$, respectively, on CIFAR-10. We confirm the efficacy of $\texttt{Q3R}$ on Transformers across both vision and language tasks, including low-rank fine-tuning.