FedWon: Triumphing Multi-domain Federated Learning Without Normalization

TL;DR

FedWon introduces a normalization-free federated learning method that effectively handles multi-domain data heterogeneity, outperforming existing approaches in accuracy and robustness across various datasets and models.

Contribution

The paper proposes FedWon, a novel normalization-free federated learning approach that addresses multi-domain data challenges by reparameterizing convolution layers, improving performance over state-of-the-art methods.

Findings

FedWon surpasses FedAvg and FedBN in accuracy across all tested datasets.

FedWon maintains strong performance even with small batch sizes.

FedWon effectively handles both multi-domain and skewed label distribution scenarios.

Abstract

Federated learning (FL) enhances data privacy with collaborative in-situ training on decentralized clients. Nevertheless, FL encounters challenges due to non-independent and identically distributed (non-i.i.d) data, leading to potential performance degradation and hindered convergence. While prior studies predominantly addressed the issue of skewed label distribution, our research addresses a crucial yet frequently overlooked problem known as multi-domain FL. In this scenario, clients' data originate from diverse domains with distinct feature distributions, instead of label distributions. To address the multi-domain problem in FL, we propose a novel method called Federated learning Without normalizations (FedWon). FedWon draws inspiration from the observation that batch normalization (BN) faces challenges in effectively modeling the statistics of multiple domains, while existing normalization techniques possess their own limitations. In order to address these issues, FedWon eliminates the normalization layers in FL and reparameterizes convolution layers with scaled weight standardization. Through extensive experimentation on five datasets and five models, our comprehensive experimental results demonstrate that FedWon surpasses both FedAvg and the current state-of-the-art method (FedBN) across all experimental setups, achieving notable accuracy improvements of more than 10% in certain domains. Furthermore, FedWon is versatile for both cross-silo and cross-device FL, exhibiting robust domain generalization capability, showcasing strong performance even with a batch size as small as 1, thereby catering to resource-constrained devices. Additionally, FedWon can also effectively tackle the challenge of skewed label distribution.