Investigating Neuron Disturbing in Fusing Heterogeneous Neural Networks

TL;DR

This paper uncovers the phenomenon of neuron disturbing in federated learning model fusion, providing a Bayesian explanation and proposing an adaptive method (AMS) that improves robustness across heterogeneous models.

Contribution

It introduces the concept of neuron disturbing, offers a Bayesian analysis, and proposes AMS, an adaptive fusion method that handles heterogeneity better than existing approaches.

Findings

AMS outperforms traditional fusion methods in heterogeneous data scenarios

Neuron disturbing significantly impacts model fusion effectiveness

AMS enables fusion across models with different architectures

Abstract

Fusing deep learning models trained on separately located clients into a global model in a one-shot communication round is a straightforward implementation of Federated Learning. Although current model fusion methods are shown experimentally valid in fusing neural networks with almost identical architectures, they are rarely theoretically analyzed. In this paper, we reveal the phenomenon of neuron disturbing, where neurons from heterogeneous local models interfere with each other mutually. We give detailed explanations from a Bayesian viewpoint combining the data heterogeneity among clients and properties of neural networks. Furthermore, to validate our findings, we propose an experimental method that excludes neuron disturbing and fuses neural networks via adaptively selecting a local model, called AMS, to execute the prediction according to the input. The experiments demonstrate that AMS is more robust in data heterogeneity than general model fusion and ensemble methods. This implies the necessity of considering neural disturbing in model fusion. Besides, AMS is available for fusing models with varying architectures as an experimental algorithm, and we also list several possible extensions of AMS for future work.