FAQS: Communication-efficient Federate DNN Architecture and Quantization Co-Search for personalized Hardware-aware Preferences

TL;DR

FAQS is a federated learning framework that significantly reduces communication costs by combining neural architecture search and quantization, enabling personalized, hardware-aware DNN models with minimal message transmission.

Contribution

The paper introduces FAQS, a novel federated learning approach that integrates NAS and quantization to lower communication costs and support personalized, hardware-aware DNN models.

Findings

Achieves 1.58x reduction in communication bandwidth per round.

Achieves 4.51x reduction compared to FL+NAS.

Supports heterogeneous, hardware-aware model customization.

Abstract

Due to user privacy and regulatory restrictions, federate learning (FL) is proposed as a distributed learning framework for training deep neural networks (DNN) on decentralized data clients. Recent advancements in FL have applied Neural Architecture Search (NAS) to replace the predefined one-size-fit-all DNN model, which is not optimal for all tasks of various data distributions, with searchable DNN architectures. However, previous methods suffer from expensive communication cost rasied by frequent large model parameters transmission between the server and clients. Such difficulty is further amplified when combining NAS algorithms, which commonly require prohibitive computation and enormous model storage. Towards this end, we propose FAQS, an efficient personalized FL-NAS-Quantization framework to reduce the communication cost with three features: weight-sharing super kernels, bit-sharing quantization and masked transmission. FAQS has an affordable search time and demands very limited size of transmitted messages at each round. By setting different personlized pareto function loss on local clients, FAQS can yield heterogeneous hardware-aware models for various user preferences. Experimental results show that FAQS achieves average reduction of 1.58x in communication bandwith per round compared with normal FL framework and 4.51x compared with FL+NAS framwork.