Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixtures

TL;DR

This paper proves that gradient EM algorithm globally converges for over-parameterized Gaussian Mixture Models with more than two components, under mild over-parameterization, using novel analytical tools.

Contribution

It provides the first global convergence guarantee for gradient EM in over-parameterized GMMs with more than two components, advancing theoretical understanding.

Findings

Gradient EM converges globally for n=Ω(m log m) components.

Convergence occurs at a polynomial rate with polynomial samples.

Introduces new analytical tools using Hermite polynomials and tensor decomposition.

Abstract

Learning Gaussian Mixture Models (GMMs) is a fundamental problem in machine learning, with the Expectation-Maximization (EM) algorithm and its popular variant gradient EM being arguably the most widely used algorithms in practice. In the exact-parameterized setting, where both the ground truth GMM and the learning model have the same number of components $m$, a vast line of work has aimed to establish rigorous recovery guarantees for EM. However, global convergence has only been proven for the case of $m=2$, and EM is known to fail to recover the ground truth when $m\geq 3$. In this paper, we consider the $\textit{over-parameterized}$ setting, where the learning model uses $n>m$ components to fit an $m$-component ground truth GMM. In contrast to the exact-parameterized case, we provide a rigorous global convergence guarantee for gradient EM. Specifically, for any well separated GMMs in general position, we prove that with only mild over-parameterization $n = \Omega(m\log m)$, randomly initialized gradient EM converges globally to the ground truth at a polynomial rate with polynomial samples. Our analysis proceeds in two stages and introduces a suite of novel tools for Gaussian Mixture analysis. We use Hermite polynomials to study the dynamics of gradient EM and employ tensor decomposition to characterize the geometric landscape of the likelihood loss. This is the first global convergence and recovery result for EM or Gradient EM beyond the special case of $m=2$.