Decentralized Low-Rank Fine-Tuning of Large Language Models

TL;DR

Dec-LoRA introduces a decentralized fine-tuning method for large language models using Low-Rank Adaptation, enabling privacy-preserving, scalable training without centralized data aggregation, and demonstrates comparable performance to centralized approaches.

Contribution

This work presents Dec-LoRA, the first decentralized algorithm for LLM fine-tuning based on LoRA, with theoretical convergence guarantees and practical effectiveness in distributed settings.

Findings

Dec-LoRA achieves performance comparable to centralized LoRA.

The algorithm converges to a stationary point for non-convex, smooth loss functions.

Dec-LoRA is effective under data heterogeneity and quantization constraints.

Abstract

While parameter-efficient fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) offer computationally efficient adaptations of Large Language Models (LLMs), their practical deployment often assumes centralized data and training environments. However, real-world scenarios frequently involve distributed, privacy-sensitive datasets that require decentralized solutions. Federated learning (FL) addresses data privacy by coordinating model updates across clients, but it is typically based on centralized aggregation through a parameter server, which can introduce bottlenecks and communication constraints. Decentralized learning, in contrast, eliminates this dependency by enabling direct collaboration between clients, improving scalability and efficiency in distributed environments. Despite its advantages, decentralized LLM fine-tuning remains underexplored. In this work, we propose Dec-LoRA, a decentralized fine-tuning algorithm for LLMs based on LoRA. Through extensive experiments on BERT and LLaMA-2 models, we demonstrate that Dec-LoRA achieves performance comparable to centralized LoRA under various conditions, including data heterogeneity and quantization constraints. Additionally, we provide a rigorous theoretical guarantee proving the convergence of our algorithm to a stationary point for non-convex and smooth loss functions. These findings highlight the potential of Dec-LoRA for scalable LLM fine-tuning in decentralized environments.