Stochastic First-Order Learning for Large-Scale Flexibly Tied Gaussian Mixture Model

TL;DR

This paper introduces a stochastic first-order optimization method on the manifold of orthogonal matrices to efficiently fit large-scale, high-dimensional flexibly-tied Gaussian Mixture Models, outperforming traditional EM algorithms.

Contribution

It proposes a novel stochastic optimization algorithm tailored for flexibly-tied GMMs on the orthogonal matrix manifold, addressing scalability and efficiency issues.

Findings

Stochastic methods achieve higher likelihoods than EM.

Fewer epochs needed for convergence with stochastic methods.

Reduced computational time per epoch with stochastic optimization.

Abstract

Gaussian Mixture Models (GMMs) are one of the most potent parametric density models used extensively in many applications. Flexibly-tied factorization of the covariance matrices in GMMs is a powerful approach for coping with the challenges of common GMMs when faced with high-dimensional data and complex densities which often demand a large number of Gaussian components. However, the expectation-maximization algorithm for fitting flexibly-tied GMMs still encounters difficulties with streaming and very large dimensional data. To overcome these challenges, this paper suggests the use of first-order stochastic optimization algorithms. Specifically, we propose a new stochastic optimization algorithm on the manifold of orthogonal matrices. Through numerous empirical results on both synthetic and real datasets, we observe that stochastic optimization methods can outperform the expectation-maximization algorithm in terms of attaining better likelihood, needing fewer epochs for convergence, and consuming less time per each epoch.