Robust High Dimensional Expectation Maximization Algorithm via Trimmed Hard Thresholding

TL;DR

This paper introduces a robust high-dimensional EM algorithm with trimming and hard thresholding steps, capable of handling arbitrarily corrupted samples in sparse latent variable models, with proven convergence and applicability to multiple models.

Contribution

It proposes a novel robust EM algorithm with trimming and hard thresholding, providing theoretical guarantees and broad applicability to high-dimensional corrupted data scenarios.

Findings

Algorithm converges geometrically under mild conditions.

Effective in models like Gaussian mixtures, regressions, and missing covariates.

Supports high corruption levels up to (rac{1}{\u221a{n}}).

Abstract

In this paper, we study the problem of estimating latent variable models with arbitrarily corrupted samples in high dimensional space ({\em i.e.,} $d\gg n$) where the underlying parameter is assumed to be sparse. Specifically, we propose a method called Trimmed (Gradient) Expectation Maximization which adds a trimming gradients step and a hard thresholding step to the Expectation step (E-step) and the Maximization step (M-step), respectively. We show that under some mild assumptions and with an appropriate initialization, the algorithm is corruption-proofing and converges to the (near) optimal statistical rate geometrically when the fraction of the corrupted samples $\epsilon$ is bounded by $ \tilde{O}(\frac{1}{\sqrt{n}})$. Moreover, we apply our general framework to three canonical models: mixture of Gaussians, mixture of regressions and linear regression with missing covariates. Our theory is supported by thorough numerical results.