Movable Antenna Enhanced Federated Fine-Tuning of Large Language Models via Hybrid Client Selection Optimization

TL;DR

This paper introduces a movable-antenna array system for federated fine-tuning of large language models over bandwidth-limited 6G links, optimizing client selection and antenna placement to improve accuracy and fairness.

Contribution

It proposes a novel joint optimization framework for antenna placement, client selection, and resource allocation in federated LLM fine-tuning using analog OTA aggregation.

Findings

Achieves up to 54.4% lower perplexity compared to non-MA baselines.

Improves participation fairness across different uplink concurrency levels.

Reduces end-to-end latency and energy consumption in federated LLM training.

Abstract

Federated fine-tuning of large language models (LLMs) over bandwidth-limited 6G links must meet strict round-time and energy budgets. Analog over-the-air (OTA) aggregation reduces uplink cost but is sensitive to fading and interference, which distort the aggregated gradient. We consider a two-phase workflow (centralized pre-training followed by federated fine-tuning) where the base station uses a movable-antenna (MA) array. In each round, MA element positions and the receive/transmit beamformers are adjusted under minimum-spacing constraints to reshape the channel and improve OTA aggregation without increasing user transmit power. We formulate a mixed-integer, nonconvex resource-allocation problem that jointly selects clients and optimizes the number of global rounds, CPU frequencies, mini-batch sizes, MA positions, and analog weights under end-to-end latency and energy limits. A successive convex approximation-penalty dual decomposition (SCA-PDD) routine alternates convex updates with oblique-manifold beamforming and spacing-aware MA placement. Experiments on OpenLLaMA-v2 (3B) with LoRA and 4-bit quantization on Alpaca and Dolly (10 clients) attain round-30 validation perplexities as low as 2.94 (Alpaca, K=1) and 4.62 (Dolly, K=1). Relative to the strongest non-MA baseline at the same concurrency, this corresponds to 17.4 percent (Alpaca, K=1) and 54.4 percent (Dolly, K=1) lower perplexity; at K=2 the reductions are 14.2 percent (Alpaca) and 13.7 percent (Dolly). Participation fairness also improves across all uplink concurrencies K in {1,2,4,8}, with the largest margins when fewer clients transmit per round.