FedSPU: Personalized Federated Learning for Resource-constrained Devices with Stochastic Parameter Update

TL;DR

FedSPU introduces a stochastic parameter update method for personalized federated learning on resource-constrained devices, outperforming dropout strategies by improving accuracy and reducing training time through neuron freezing and early stopping.

Contribution

The paper proposes FedSPU, a novel method that maintains full models with stochastic neuron freezing, enhancing robustness and efficiency in resource-limited federated learning environments.

Findings

FedSPU outperforms federated dropout by 7.57% in accuracy.

Early stopping reduces training time by up to 70.4%.

FedSPU improves model robustness against biased local data.

Abstract

Personalized Federated Learning (PFL) is widely employed in IoT applications to handle high-volume, non-iid client data while ensuring data privacy. However, heterogeneous edge devices owned by clients may impose varying degrees of resource constraints, causing computation and communication bottlenecks for PFL. Federated Dropout has emerged as a popular strategy to address this challenge, wherein only a subset of the global model, i.e. a sub-model, is trained on a client's device, thereby reducing computation and communication overheads. Nevertheless, the dropout-based model-pruning strategy may introduce bias, particularly towards non-iid local data. When biased sub-models absorb highly divergent parameters from other clients, performance degradation becomes inevitable. In response, we propose federated learning with stochastic parameter update (FedSPU). Unlike dropout that tailors the global model to small-size local sub-models, FedSPU maintains the full model architecture on each device but randomly freezes a certain percentage of neurons in the local model during training while updating the remaining neurons. This approach ensures that a portion of the local model remains personalized, thereby enhancing the model's robustness against biased parameters from other clients. Experimental results demonstrate that FedSPU outperforms federated dropout by 7.57% on average in terms of accuracy. Furthermore, an introduced early stopping scheme leads to a significant reduction of the training time by 24.8%-70.4% while maintaining high accuracy.