Statistical guarantees for the EM algorithm: From population to sample-based analysis

TL;DR

This paper provides a rigorous theoretical framework for analyzing the EM algorithm's performance, establishing guarantees from population to finite-sample scenarios across common incomplete-data models.

Contribution

It introduces a novel analysis of EM and gradient EM algorithms, characterizing their attraction domains and providing non-asymptotic guarantees for finite samples.

Findings

Guarantees on EM convergence with high probability given proper initialization.

Theoretical bounds for the number of steps needed to reach statistical accuracy.

Validation of theory through simulations on canonical models.

Abstract

We develop a general framework for proving rigorous guarantees on the performance of the EM algorithm and a variant known as gradient EM. Our analysis is divided into two parts: a treatment of these algorithms at the population level (in the limit of infinite data), followed by results that apply to updates based on a finite set of samples. First, we characterize the domain of attraction of any global maximizer of the population likelihood. This characterization is based on a novel view of the EM updates as a perturbed form of likelihood ascent, or in parallel, of the gradient EM updates as a perturbed form of standard gradient ascent. Leveraging this characterization, we then provide non-asymptotic guarantees on the EM and gradient EM algorithms when applied to a finite set of samples. We develop consequences of our general theory for three canonical examples of incomplete-data problems: mixture of Gaussians, mixture of regressions, and linear regression with covariates missing completely at random. In each case, our theory guarantees that with a suitable initialization, a relatively small number of EM (or gradient EM) steps will yield (with high probability) an estimate that is within statistical error of the MLE. We provide simulations to confirm this theoretically predicted behavior.