ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning

TL;DR

This paper introduces ADF-LoRA, a novel decentralized federated fine-tuning method that synchronizes low-rank updates to improve stability and convergence in peer-to-peer settings, outperforming existing variants.

Contribution

It proposes a new ADF-LoRA approach that synchronizes one low-rank matrix per round and mixes matrices to enhance stability in decentralized federated learning.

Findings

Achieves faster and smoother convergence.

Delivers highest average accuracy across multiple tasks.

Outperforms existing LoRA variants in decentralized FL.

Abstract

This paper revisits alternating low-rank updates for federated fine-tuning and examines their behavior in decentralized federated learning (DFL). While alternating the LoRA matrices has been shown to stabilize aggregation in centralized FL, extending this mechanism to decentralized, peer-to-peer communication introduces new challenges due to phase-state mismatch and block-wise divergence across clients. We introduce ADF-LoRA, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation. This design preserves the cross-term suppression effect of alternating updates while improving stability in serverless topologies. We provide a convergence analysis under standard smoothness assumptions and evaluate ADF-LoRA on multiple GLUE tasks. Experiments show that ADF-LoRA achieves faster and smoother convergence and delivers the highest average accuracy across tasks, outperforming existing LoRA variants in decentralized FL by a consistent margin.