Optimizing Client Participation in Communication-Constrained Federated LLM Adaptation with LoRA

TL;DR

This paper introduces a framework to optimize client participation in federated learning for large language models, reducing communication costs while maintaining accuracy.

Contribution

The novel contribution is a communication-aware framework, LoRaC-GA, using a genetic algorithm to dynamically select clients under bandwidth constraints.

Findings

LoRaC-GA adaptively selects the optimal number of clients per round under fixed bandwidth constraints.

The framework achieves competitive accuracy while significantly reducing communication costs.

A structured peer-to-peer protocol enables scalable communication with log2K complexity.

Abstract

Federated learning (FL) enables privacy-preserving adaptation of large language models (LLMs) across distributed clients. However, deploying FL in edge environments remains challenging because of the high communication overhead of full-model updates. Recent advances in parameter-efficient fine-tuning (PEFT), particularly low-rank adaptation (LoRA), have substantially reduced update sizes by injecting lightweight trainable matrices into pretrained transformers, thereby making FL with LLMs more feasible. In this paper, we propose LoRaC-GA, a communication-aware optimization framework that dynamically determines the optimal number of clients to participate in each round under a fixed bandwidth constraint. We formulated a max-min objective to jointly maximize the model accuracy and communication efficiency and solved the resulting non-convex problem using a genetic algorithm (GA). To further reduce the overhead, we integrated a structured peer-to-peer collaboration protocol with log2K complexity, enabling scalable communication without full connectivity. The simulation results demonstrate that LoRaC-GA adaptively selects the optimal client count, achieving competitive accuracy while significantly reducing the communication cost. The proposed framework is well-suited for bandwidth-constrained edge deployments involving large-scale LLMs.

Figures