SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning

TL;DR

SPATL is a federated learning method that reduces communication costs, accelerates inference, and improves training stability by selecting salient parameters, splitting models for heterogeneity, and controlling gradients.

Contribution

It introduces a salient parameter selection, model splitting for heterogeneous clients, and gradient control, advancing federated learning efficiency and robustness.

Findings

Reduces communication cost by up to 86.45%

Speeds up local inference by up to 39.7% FLOPs

Requires 7.4 times less communication overhead for ResNet-20

Abstract

Federated learning~(FL) facilitates the training and deploying AI models on edge devices. Preserving user data privacy in FL introduces several challenges, including expensive communication costs, limited resources, and data heterogeneity. In this paper, we propose SPATL, an FL method that addresses these issues by: (a) introducing a salient parameter selection agent and communicating selected parameters only; (b) splitting a model into a shared encoder and a local predictor, and transferring its knowledge to heterogeneous clients via the locally customized predictor. Additionally, we leverage a gradient control mechanism to further speed up model convergence and increase robustness of training processes. Experiments demonstrate that SPATL reduces communication overhead, accelerates model inference, and enables stable training processes with better results compared to state-of-the-art methods. Our approach reduces communication cost by up to $86.45\%$, accelerates local inference by reducing up to $39.7\%$ FLOPs on VGG-11, and requires $7.4 \times$ less communication overhead when training ResNet-20.