Network EM Algorithm for Gaussian Mixture Model in Decentralized Federated Learning

TL;DR

This paper introduces novel network EM algorithms for Gaussian mixture models in decentralized federated learning, improving estimation accuracy and convergence in heterogeneous and poorly-separated data scenarios.

Contribution

It proposes the momentum network EM (MNEM) and semi-supervised MNEM algorithms, with theoretical guarantees and enhanced convergence in challenging data conditions.

Findings

MNEM achieves statistical efficiency similar to full sample estimators.

semi-MNEM improves convergence speed and handles poorly-separated components.

Algorithms perform well in simulations and real data experiments.

Abstract

We systematically study various network Expectation-Maximization (EM) algorithms for the Gaussian mixture model within the framework of decentralized federated learning. Our theoretical investigation reveals that directly extending the classical decentralized supervised learning method to the EM algorithm exhibits poor estimation accuracy with heterogeneous data across clients and struggles to converge numerically when Gaussian components are poorly-separated. To address these issues, we propose two novel solutions. First, to handle heterogeneous data, we introduce a momentum network EM (MNEM) algorithm, which uses a momentum parameter to combine information from both the current and historical estimators. Second, to tackle the challenge of poorly-separated Gaussian components, we develop a semi-supervised MNEM (semi-MNEM) algorithm, which leverages partially labeled data. Rigorous theoretical analysis demonstrates that MNEM can achieve statistical efficiency comparable to that of the whole sample estimator when the mixture components satisfy certain separation conditions, even in heterogeneous scenarios. Moreover, the semi-MNEM estimator enhances the convergence speed of the MNEM algorithm, effectively addressing the numerical convergence challenges in poorly-separated scenarios. Extensive simulation and real data analyses are conducted to justify our theoretical findings.