Convergence of the EM Algorithm for Gaussian Mixtures with Unbalanced Mixing Coefficients

TL;DR

This paper investigates how unbalanced mixing coefficients affect EM algorithm convergence in Gaussian mixtures, revealing slower convergence with increased coefficient disparity, and proposes an anti-annealing method to improve speed.

Contribution

It introduces a deterministic anti-annealing algorithm that accelerates EM convergence for Gaussian mixtures with unbalanced mixing coefficients and extends this approach to Dirichlet process mixtures.

Findings

Anti-annealing significantly speeds up EM convergence.

The method outperforms BFGS, Conjugate Gradient, and traditional EM.

Advantages over variational Bayesian methods in Dirichlet mixtures.

Abstract

The speed of convergence of the Expectation Maximization (EM) algorithm for Gaussian mixture model fitting is known to be dependent on the amount of overlap among the mixture components. In this paper, we study the impact of mixing coefficients on the convergence of EM. We show that when the mixture components exhibit some overlap, the convergence of EM becomes slower as the dynamic range among the mixing coefficients increases. We propose a deterministic anti-annealing algorithm, that significantly improves the speed of convergence of EM for such mixtures with unbalanced mixing coefficients. The proposed algorithm is compared against other standard optimization techniques like BFGS, Conjugate Gradient, and the traditional EM algorithm. Finally, we propose a similar deterministic anti-annealing based algorithm for the Dirichlet process mixture model and demonstrate its advantages over the conventional variational Bayesian approach.