FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA

TL;DR

FedRot-LoRA introduces a method to align client updates via orthogonal transformations in federated LoRA, reducing aggregation errors caused by rotational misalignment and improving training stability and performance.

Contribution

It proposes a novel rotational alignment technique for federated LoRA that enhances update aggregation without extra communication costs.

Findings

FedRot-LoRA outperforms existing federated LoRA methods across various tasks.

Rotational alignment reduces aggregation error and stabilizes training.

Theoretical analysis shows tighter error bounds with alignment.

Abstract

Federated LoRA provides a communication-efficient mechanism for fine-tuning large language models on decentralized data. In practice, however, a discrepancy between the factor-wise averaging used to preserve low rank and the mathematically correct aggregation of local updates can cause significant aggregation error and unstable training. We argue that a major source of this problem is rotational misalignment, arising from the rotational invariance of low-rank factorizations -- semantically equivalent updates can be represented in different latent subspaces across clients since $(B_i R_i)(R_i^\top A_i) = B_i A_i$. When such misaligned factors are averaged directly, they interfere destructively and degrade the global update. To address this issue, we propose FedRot-LoRA, a federated LoRA framework that aligns client updates via orthogonal transformations prior to aggregation. This alignment preserves the semantic update while reducing cross-client subspace mismatch, without increasing communication cost or restricting model expressivity. We provide a convergence analysis that examines the aggregation error induced by factor-wise averaging and shows how rotational alignment yields a tighter upper bound on this error. Extensive experiments on natural language understanding and generative tasks demonstrate that FedRot-LoRA consistently outperforms existing federated LoRA baselines across a range of heterogeneity levels and LoRA ranks.