Low-Latency Federated Fine-Tuning for Large Language Models Over Wireless Networks

TL;DR

This paper introduces a joint client-specific pruning and bandwidth allocation framework to enable low-latency federated fine-tuning of large language models over wireless networks, addressing resource constraints and network variability.

Contribution

It proposes a novel optimization framework combining pruning and bandwidth allocation to minimize fine-tuning latency in federated LLMs over wireless channels.

Findings

Significantly reduces fine-tuning time compared to baselines.

Achieves comparable or better test loss with lower resource overhead.

Effective in heterogeneous client environments.

Abstract

Recently, federated large language models (LLMs) have drawn significant attention thanks to coupled capabilities of LLMs and federated learning (FL) that address privacy concerns in collaborative fine-tuning. However, due to large-scale parameters of LLMs, existing federated LLM fine-tuning frameworks incur significant challenges in resource-constrained clients characterized by heterogeneous computing capabilities and random wireless channels. To address this issue, we propose a joint client-specific pruning and bandwidth allocation (JCPBA) framework for federated LLMs to improve the fine-tuning efficiency over the wireless networks. Specifically, we formulate a fine-tuning latency minimization problem by jointly optimizing pruning rates and bandwidth allocations. Furthermore, we solve this optimization problem using a block coordinate descent method. Extensive experiments on the datasets of Yahoo Answers and GSM8K demonstrate that the proposed framework significantly reduces wall-clock fine-tuning time compared with state-of-the-art baselines and gains equal or lower test loss at the cost of lower computation and communication overhead.