Fair Marriage Principle and Initialization Map for the EM Algorithm

TL;DR

This paper challenges existing EM convergence theory, reveals factors affecting convergence speed, and introduces the Channel Matching EM algorithm with an initialization map to improve global convergence in mixture models.

Contribution

It provides new insights into EM convergence, debunks the popular theory, and proposes an improved algorithm with an initialization map for better convergence speed.

Findings

The popular convergence theory of EM is incorrect.

Unfair competition between components reduces convergence speed.

The Channel Matching EM algorithm accelerates global convergence.

Abstract

The popular convergence theory of the EM algorithm explains that the observed incomplete data log-likelihood L and the complete data log-likelihood Q are positively correlated, and we can maximize L by maximizing Q. The Deterministic Annealing EM (DAEM) algorithm was hence proposed for avoiding locally maximal Q. This paper provides different conclusions: 1) The popular convergence theory is wrong; 2) The locally maximal Q can affect the convergent speed, but cannot block the global convergence; 3) Like marriage competition, unfair competition between two components may vastly decrease the globally convergent speed; 4) Local convergence exists because the sample is too small, and unfair competition exists; 5) An improved EM algorithm, called the Channel Matching (CM) EM algorithm, can accelerate the global convergence. This paper provides an initialization map with two means as two axes for the example of a binary Gaussian mixture studied by the authors of DAEM algorithm. This map can tell how fast the convergent speeds are for different initial means and why points in some areas are not suitable as initial points. A two-dimensional example indicates that the big sample or the fair initialization can avoid global convergence. For more complicated mixture models, we need further study to convert the fair marriage principle to specific methods for the initializations.