Three Birds, One Stone: Solving the Communication-Memory-Privacy Trilemma in LLM Fine-tuning Over Wireless Networks with Zeroth-Order Optimization
Zhijie Cai, Yuhao Zheng, Haolong Chen, Dongzhu Liu, Bin Wang, Guangxu Zhu

TL;DR
This paper introduces pAirZero, a framework combining Zeroth-Order optimization and Over-the-Air computation to enable privacy-preserving, memory-efficient, and low-communication federated fine-tuning of large language models over wireless networks.
Contribution
It proposes a novel resource-efficient framework that reduces communication, memory, and privacy risks in federated LLM fine-tuning using innovative optimization and transmission techniques.
Findings
pAirZero achieves 25% peak memory cost on OPT-125M.
It significantly reduces communication load compared to traditional methods.
The framework ensures consistent privacy protection under varying channel conditions.
Abstract
Federated Learning (FL) offers a promising pathway for collaboratively fine-tuning Large Language Models (LLMs) at the edge; however, this paradigm faces a critical bottleneck: the prohibitive communication and memory overheads incurred by exchanging high-dimensional gradients. Furthermore, recent studies reveal that user training data can still be recovered from these local gradients, undermining the core privacy promise of FL. In this paper, we address this trilemma of communication, memory, and privacy by proposing pAirZero, a novel framework that synergizes Zeroth-Order (ZO) optimization with Over-the-Air (OTA) computation. Uniquely, pAirZero enables resource-constrained devices to submit their local gradient with only bit-level communication loads while participating in federated fine-tuning of LLMs with inference-level memory costs. This approach not only eliminates the high…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
