Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

TL;DR

This paper introduces LA-LoRA, a novel method for privacy-preserving federated learning that improves fine-tuning of large models under differential privacy constraints by addressing gradient coupling and noise amplification issues.

Contribution

LA-LoRA decouples gradient interactions and aligns updates across clients, enhancing robustness and convergence in privacy-preserving federated fine-tuning of large models.

Findings

LA-LoRA outperforms baseline methods like RoLoRA by 16.83% in accuracy on Tiny-ImageNet.

LA-LoRA achieves state-of-the-art results on Swin Transformer and RoBERTa models under DP constraints.

Theoretical analysis confirms improved convergence guarantees with LA-LoRA.

Abstract

Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private federated learning (DPFL) is hindered by a fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to performance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous update of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model in the parameter space. To address these issues, we propose LA-LoRA (\textbf{L}ocal \textbf{A}lternating \textbf{LoRA}), a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA-LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example, when fine-tuning the Swin-B model on the Tiny-ImageNet dataset under a strict privacy budget ($\epsilon = 1$), LA-LoRA outperforms the best baseline, RoLoRA, by 16.83\% in test accuracy. Code is provided in \repolink.