Learning Adapter Rank via Symmetry Breaking

TL;DR

This paper introduces BayesLoRA, a Bayesian low-rank adaptation method that automatically determines effective adapter ranks and improves uncertainty estimation by breaking rotational symmetry in low-rank models.

Contribution

It proposes a variational inference framework that resolves non-identifiability in low-rank adapters, enabling automatic relevance determination and better uncertainty calibration.

Findings

BayesLoRA learns effective adapter ranks aligned with dominant singular directions.

It achieves stable rank structures and improves predictive calibration.

Matches or exceeds performance of low-rank sparsification baselines at similar cost.

Abstract

Low-rank adaptation is effective partly because downstream updates lie in a low-dimensional subspace, but the latent rank coordinates of LoRA are not identifiable: any invertible reparameterization of the adapter factors leaves the weight update unchanged. We show that variational inference with a diagonal rank-wise posterior turns this non-identifiability into a useful inductive bias. By breaking LoRA's rotational gauge symmetry, the variational objective selects a preferred basis in rank space, enabling automatic relevance determination over rank directions. This yields Low-Rank Variational Dropout (LRVD), a Bayesian framework that performs inference directly in the low-rank adaptation space rather than the ambient weight space. As an instantiation, BayesLoRA jointly learns effective adapter rank and predictive uncertainty with only $\mathcal{O}(r)$ additional parameters. Empirically, BayesLoRA induces stable rank structure aligned with the dominant singular directions of learned updates, yields compact predictive calibration and matches or exceeds strong low-rank sparsification baselines at comparable training cost.