A Method of Moments for Mixture Models and Hidden Markov Models

TL;DR

This paper introduces an efficient method of moments approach for high-dimensional mixture models and hidden Markov models, providing a practical and theoretically sound alternative to traditional EM algorithms.

Contribution

It develops a novel method of moments technique that scales efficiently and achieves rigorous learning guarantees for complex mixture models and HMMs.

Findings

Provides a scalable method with polynomial sample complexity

Achieves consistent parameter estimation for high-dimensional mixtures

Offers a practical alternative to EM algorithms

Abstract

Mixture models are a fundamental tool in applied statistics and machine learning for treating data taken from multiple subpopulations. The current practice for estimating the parameters of such models relies on local search heuristics (e.g., the EM algorithm) which are prone to failure, and existing consistent methods are unfavorable due to their high computational and sample complexity which typically scale exponentially with the number of mixture components. This work develops an efficient method of moments approach to parameter estimation for a broad class of high-dimensional mixture models with many components, including multi-view mixtures of Gaussians (such as mixtures of axis-aligned Gaussians) and hidden Markov models. The new method leads to rigorous unsupervised learning results for mixture models that were not achieved by previous works; and, because of its simplicity, it offers a viable alternative to EM for practical deployment.