FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantization

TL;DR

FedWSQ introduces a novel federated learning framework that combines weight standardization and distribution-aware non-uniform quantization to improve robustness and reduce communication costs under data heterogeneity.

Contribution

The paper proposes FedWSQ, integrating weight standardization and a new quantization method to enhance federated learning performance and efficiency.

Findings

Outperforms existing FL methods in accuracy.

Reduces communication overhead significantly.

Effective under extreme data heterogeneity.

Abstract

Federated learning (FL) often suffers from performance degradation due to key challenges such as data heterogeneity and communication constraints. To address these limitations, we present a novel FL framework called FedWSQ, which integrates weight standardization (WS) and the proposed distribution-aware non-uniform quantization (DANUQ). WS enhances FL performance by filtering out biased components in local updates during training, thereby improving the robustness of the model against data heterogeneity and unstable client participation. In addition, DANUQ minimizes quantization errors by leveraging the statistical properties of local model updates. As a result, FedWSQ significantly reduces communication overhead while maintaining superior model accuracy. Extensive experiments on FL benchmark datasets demonstrate that FedWSQ consistently outperforms existing FL methods across various challenging FL settings, including extreme data heterogeneity and ultra-low-bit communication scenarios.