Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

TL;DR

The paper introduces AS-LoRA, an adaptive framework for privacy-preserving federated learning that dynamically selects LoRA components to improve accuracy and convergence without extra privacy costs.

Contribution

It proposes a novel adaptive method that considers layer-wise and round-wise dynamics, along with a curvature-aware score, to enhance federated LoRA training.

Findings

AS-LoRA improves accuracy by up to 7.5 percentage points on GLUE.

It accelerates convergence and biases solutions toward flatter minima.

Achieves comparable or better results than SVD-based methods with much lower cost.

Abstract

Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure, which is further amplified by DP noise and degrades both stability and accuracy. Existing remedies apply a single update mode uniformly across all layers and all communication rounds (or alternate them on a fixed schedule), ignoring both the structural asymmetry between the two LoRA factors and the round-wise dynamics of training. We propose AS-LoRA, an adaptive framework defined by three axes (i) layer-wise freedom, in which each layer independently selects its active component, (ii) round-wise adaptivity, in which the selection updates over communication rounds, and (iii) a curvature-aware score derived from a second-order approximation of the loss. Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost. Across GLUE, SQuAD, CIFAR-100, and Tiny-ImageNet under strict DP budgets and non-IID partitions, AS-LoRA improves over the federated LoRA baselines by up to $+7.5$ pp on GLUE and $+12.5$ pp on MNLI-mm for example, while matching or exceeding SVD-based aggregation methods at $33\text{--}180 \times$ lower aggregation cost and with negligible communication overhead. Code for the proposed method is available at https://anonymous.4open.science/r/as_lora-F75F/.