The Key of Parameter Skew in Federated Learning

TL;DR

This paper identifies the issue of parameter skew in federated learning caused by data heterogeneity, and proposes FedSA, an aggregation method that improves global model accuracy by addressing parameter dispersion.

Contribution

The paper introduces the concept of parameter skew in federated learning and proposes FedSA, a novel aggregation strategy that categorizes parameters to enhance model performance.

Findings

FedSA outperforms eight baselines by approximately 4.7% in test accuracy.

Parameter skew significantly impacts the accuracy of federated learning models.

FedSA effectively mitigates the effects of parameter skew across different datasets.

Abstract

Federated Learning (FL) has emerged as an excellent solution for performing deep learning on different data owners without exchanging raw data. However, statistical heterogeneity in FL presents a key challenge, leading to a phenomenon of skewness in local model parameter distributions that researchers have largely overlooked. In this work, we propose the concept of parameter skew to describe the phenomenon that can substantially affect the accuracy of global model parameter estimation. Additionally, we introduce FedSA, an aggregation strategy to obtain a high-quality global model, to address the implication from parameter skew. Specifically, we categorize parameters into high-dispersion and low-dispersion groups based on the coefficient of variation. For high-dispersion parameters, Micro-Classes (MIC) and Macro-Classes (MAC) represent the dispersion at the micro and macro levels, respectively, forming the foundation of FedSA. To evaluate the effectiveness of FedSA, we conduct extensive experiments with different FL algorithms on three computer vision datasets. FedSA outperforms eight state-of-the-art baselines by about 4.7% in test accuracy.