Byzantine-tolerant distributed learning of finite mixture models

TL;DR

This paper presents DFMR, a Byzantine-tolerant method for distributed learning of finite mixture models that is both computationally efficient and statistically robust.

Contribution

It introduces Distance Filtered Mixture Reduction (DFMR), a novel algorithm that robustly aggregates local estimates in the presence of Byzantine failures.

Findings

DFMR achieves optimal convergence rate.

DFMR effectively filters out corrupted estimates.

Numerical experiments confirm robustness and accuracy.

Abstract

Traditional statistical methods need to be updated to work with modern distributed data storage paradigms. A common approach is the split-and-conquer framework, which involves learning models on local machines and averaging their parameter estimates. However, this does not work for the important problem of learning finite mixture models, because subpopulation indices on each local machine may be arbitrarily permuted (the "label switching problem"). Zhang and Chen (2022) proposed Mixture Reduction (MR) to address this issue, but MR remains vulnerable to Byzantine failure, whereby a fraction of local machines may transmit arbitrarily erroneous information. This paper introduces Distance Filtered Mixture Reduction (DFMR), a Byzantine tolerant adaptation of MR that is both computationally efficient and statistically sound. DFMR leverages the densities of local estimates to construct a robust filtering mechanism. By analysing the pairwise L2 distances between local estimates, DFMR identifies and removes severely corrupted local estimates while retaining the majority of uncorrupted ones. We provide theoretical justification for DFMR, proving its optimal convergence rate and asymptotic equivalence to the global maximum likelihood estimate under standard assumptions. Numerical experiments on simulated and real-world data validate the effectiveness of DFMR in achieving robust and accurate aggregation in the presence of Byzantine failure.