Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning

TL;DR

This paper provides a theoretical convergence analysis of aggregation strategies in LoRA-enabled federated learning, clarifying conditions for global and local model convergence and comparing their effectiveness.

Contribution

It introduces a unified convergence framework for LoRA-based federated learning, categorizes aggregation methods, and analyzes their convergence properties with theoretical guarantees.

Findings

Sum-Product aggregation satisfies weak convergence conditions.

Product-Sum aggregation satisfies strong convergence conditions.

Experimental results validate the theoretical convergence analysis.

Abstract

Federated Learning (FL) enables collaborative model training across decentralized data sources while preserving data privacy. However, the growing size of Machine Learning (ML) models poses communication and computation challenges in FL. Low-Rank Adaptation (LoRA) has recently been introduced into FL as an efficient fine-tuning method, reducing communication overhead by updating only a small number of trainable parameters. Despite its effectiveness, how to aggregate LoRA-updated local models on the server remains a critical and understudied problem. In this paper, we provide a unified convergence analysis for LoRA-based FL. We first categories the current aggregation method into two major type: Sum-Product (SP) and Product-Sum (PS). Then we formally define the Aggregation-Broadcast Operator (ABO) and derive both weak and strong convergence condition under mild assumptions. Furthermore, we present both weak and strong convergence condition that guarantee convergence of the local model and the global model respectively. These theoretical analyze offer a principled understanding of various aggregation strategies. Notably, we prove that the SP and PS aggregation methods satisfy the weak and strong convergence condition respectively, but differ in their ability to achieve the optimal convergence rate. Extensive experiments on standard benchmarks validate our theoretical findings.