Global analysis of Expectation Maximization for mixtures of two Gaussians

TL;DR

This paper provides a comprehensive global analysis of the Expectation Maximization algorithm for two-Gaussian mixture models, clarifying its convergence behavior and statistical properties in both idealized and practical scenarios.

Contribution

It offers the first detailed characterization of EM's convergence and statistical consistency for two-Gaussian mixtures, bridging the gap between theory and practice.

Findings

Characterizes the limit points of EM in the infinite sample limit.

Establishes conditions for statistical consistency of EM.

Provides insights into EM's convergence behavior based on initial parameters.

Abstract

Expectation Maximization (EM) is among the most popular algorithms for estimating parameters of statistical models. However, EM, which is an iterative algorithm based on the maximum likelihood principle, is generally only guaranteed to find stationary points of the likelihood objective, and these points may be far from any maximizer. This article addresses this disconnect between the statistical principles behind EM and its algorithmic properties. Specifically, it provides a global analysis of EM for specific models in which the observations comprise an i.i.d. sample from a mixture of two Gaussians. This is achieved by (i) studying the sequence of parameters from idealized execution of EM in the infinite sample limit, and fully characterizing the limit points of the sequence in terms of the initial parameters; and then (ii) based on this convergence analysis, establishing statistical consistency (or lack thereof) for the actual sequence of parameters produced by EM.