Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models

TL;DR

FedARA introduces an adaptive rank allocation framework for federated fine-tuning of language models, improving performance, communication efficiency, and resource usage across heterogeneous devices.

Contribution

It proposes a novel adaptive rank allocation method using truncated SVD, dynamic rank adjustment, and module pruning to enhance federated parameter-efficient fine-tuning.

Findings

Outperforms baselines by 6.95% to 8.49% in accuracy.

Improves communication efficiency by 2.40×.

Reduces training time and energy consumption by up to 48.90% and 46.95%.

Abstract

Pre-trained Language Models (PLMs) have demonstrated their superiority and versatility in modern Natural Language Processing (NLP), effectively adapting to various downstream tasks through further fine-tuning. Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a promising solution to address privacy and efficiency challenges in distributed training for PLMs on resource-constrained local devices. However, our measurements reveal two key limitations of FedPEFT: heterogeneous data across devices exacerbates performance degradation of low-rank adaptation, and a fixed parameter configuration results in communication inefficiency. To overcome these limitations, we propose FedARA, a novel adaptive rank allocation framework for federated parameter-efficient fine-tuning of language models. Specifically, FedARA employs truncated Singular Value Decomposition (SVD) adaptation to enhance similar feature representation across clients, significantly mitigating the adverse effects of data heterogeneity. Subsequently, it utilizes dynamic rank allocation to progressively identify critical ranks, effectively improving communication efficiency. Lastly, it leverages rank-based module pruning to automatically remove inactive modules, steadily reducing local computational cost and memory usage in each federated learning round. Extensive experiments show that FedARA consistently outperforms baselines by an average of 6.95% to 8.49% across various datasets and models under heterogeneous data while significantly improving communication efficiency by 2.40$ \times$. Moreover, experiments on various edge devices demonstrate substantial decreases in total training time and energy consumption by up to 48.90% and 46.95%, respectively.