Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA

TL;DR

GLoRA introduces a gauge-aware server representation for federated LoRA, enabling semantically meaningful aggregation of low-rank updates across heterogeneous clients, improving performance and efficiency.

Contribution

It proposes GLoRA, a novel gauge-aware aggregation method that estimates a shared update subspace and supports rank-heterogeneous clients in federated LoRA.

Findings

GLoRA outperforms baseline federated LoRA methods on GLUE and SuperNI datasets.

It effectively handles client heterogeneity, including different ranks and sparse participation.

GLoRA achieves a better efficiency-performance trade-off compared to existing approaches.

Abstract

Federated LoRA enables parameter-efficient adaptation of large language models under decentralized data and limited client resources.However, directly averaging LoRA factors is representation-dependent: the same intrinsic update admits infinitely many gauge-equivalent factorizations, so factor-level aggregation can change under arbitrary coordinate choices while the underlying update remains unchanged. This reveals a semantic mismatch in existing federated LoRA aggregation rules. We propose \textbf{GLoRA}, a gauge-aware server representation for federated LoRA.Instead of aggregating raw factors, GLoRA estimates a consensus update subspace from client projectors and aggregates client updates in shared reference coordinates, thereby representing semantic update aggregation entirely in low-rank form. To support heterogeneous client capacities, GLoRA further provides a rank-compatible readout that instantiates adapters of different ranks from the same server state without dense update reconstruction. Experiments on GLUE and SuperNI show that GLoRA consistently outperforms federated LoRA baselines under data, resource, and task heterogeneity, including heterogeneous client ranks, sparse participation, larger backbones, and unseen-task evaluation. GLoRA also achieves a favorable efficiency--performance trade-off, suggesting that effective federated LoRA requires not merely averaging low-rank factors, but defining a semantically meaningful server-side representation for aggregation.