Sharp Analysis of Expectation-Maximization for Weakly Identifiable Models

TL;DR

This paper rigorously analyzes the slow convergence of the EM algorithm in weakly identifiable Gaussian mixture models, revealing convergence rates of order $n^{3/4}$ steps and estimation errors of order $n^{-1/8}$ and $n^{-1/4}$ in univariate cases.

Contribution

It provides the first detailed characterization of EM's slow convergence in weakly identifiable models, introducing a novel two-stage localization argument and extending results to multivariate cases.

Findings

EM converges in order $n^{3/4}$ steps in univariate models.

Estimates are within $n^{-1/8}$ and $n^{-1/4}$ of true parameters.

Similar slow rates are observed in multivariate models.

Abstract

We study a class of weakly identifiable location-scale mixture models for which the maximum likelihood estimates based on $n$ i.i.d. samples are known to have lower accuracy than the classical $n^{- \frac{1}{2}}$ error. We investigate whether the Expectation-Maximization (EM) algorithm also converges slowly for these models. We provide a rigorous characterization of EM for fitting a weakly identifiable Gaussian mixture in a univariate setting where we prove that the EM algorithm converges in order $n^{\frac{3}{4}}$ steps and returns estimates that are at a Euclidean distance of order ${ n^{- \frac{1}{8}}}$ and ${ n^{-\frac{1} {4}}}$ from the true location and scale parameter respectively. Establishing the slow rates in the univariate setting requires a novel localization argument with two stages, with each stage involving an epoch-based argument applied to a different surrogate EM operator at the population level. We demonstrate several multivariate ($d \geq 2$) examples that exhibit the same slow rates as the univariate case. We also prove slow statistical rates in higher dimensions in a special case, when the fitted covariance is constrained to be a multiple of the identity.