Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

TL;DR

This paper introduces LEGEND, a LoRA-based federated fine-tuning framework that efficiently adapts pre-trained models on heterogeneous devices by optimizing LoRA configurations, significantly improving speed and reducing communication costs.

Contribution

The paper proposes a novel LoRA configuration algorithm for federated fine-tuning that accounts for device heterogeneity, enhancing efficiency and resource utilization.

Findings

Achieves 1.5-2.8× speedup in fine-tuning.

Reduces communication costs by approximately 42.3%.

Maintains comparable accuracy with advanced solutions.

Abstract

Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8$\times$ and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.