Learning Overspecified Gaussian Mixtures Exponentially Fast with the EM Algorithm

TL;DR

This paper proves that the EM algorithm converges exponentially fast for overspecified Gaussian mixture models under certain structured conditions, providing both theoretical guarantees and practical insights.

Contribution

It establishes the exponential convergence rate of EM in overspecified Gaussian mixtures with structured means, extending to finite samples and offering practical implications.

Findings

EM converges exponentially fast in overspecified Gaussian mixtures.

Explicit statistical convergence guarantees are derived.

Numerical experiments confirm the theoretical acceleration in convergence.

Abstract

We investigate the convergence properties of the EM algorithm when applied to overspecified Gaussian mixture models -- that is, when the number of components in the fitted model exceeds that of the true underlying distribution. Focusing on a structured configuration where the component means are positioned at the vertices of a regular simplex and the mixture weights satisfy a non-degeneracy condition, we demonstrate that the population EM algorithm converges exponentially fast in terms of the Kullback-Leibler (KL) distance. Our analysis leverages the strong convexity of the negative log-likelihood function in a neighborhood around the optimum and utilizes the Polyak-{\L}ojasiewicz inequality to establish that an $\epsilon$-accurate approximation is achievable in $O(\log(1/\epsilon))$ iterations. Furthermore, we extend these results to a finite-sample setting by deriving explicit statistical convergence guarantees. Numerical experiments on synthetic datasets corroborate our theoretical findings, highlighting the dramatic acceleration in convergence compared to conventional sublinear rates. This work not only deepens the understanding of EM's behavior in overspecified settings but also offers practical insights into initialization strategies and model design for high-dimensional clustering and density estimation tasks.