Understanding and Accelerating EM Algorithm's Convergence by Fair Competition Principle and Rate-Verisimilitude Function

TL;DR

This paper introduces the Fair Competition Principle and rate-verisimilitude function to explain and accelerate EM algorithm convergence, emphasizing the importance of initializations and competition fairness, especially in small samples.

Contribution

It proposes the Fair Competition Principle and an initialization map to improve EM convergence speed and robustness, supported by a new theoretical framework based on rate-verisimilitude functions.

Findings

FCP improves initializations and convergence speed.

The initialization map significantly reduces running times.

Theoretical analysis explains EM convergence behavior.

Abstract

Why can the Expectation-Maximization (EM) algorithm for mixture models converge? Why can different initial parameters cause various convergence difficulties? The Q-L synchronization theory explains that the observed data log-likelihood L and the complete data log-likelihood Q are positively correlated; we can achieve maximum L by maximizing Q. According to this theory, the Deterministic Annealing EM (DAEM) algorithm's authors make great efforts to eliminate locally maximal Q for avoiding L's local convergence. However, this paper proves that in some cases, Q may and should decrease for L to increase; slow or local convergence exists only because of small samples and unfair competition. This paper uses marriage competition to explain different convergence difficulties and proposes the Fair Competition Principle (FCP) with an initialization map for improving initializations. It uses the rate-verisimilitude function, extended from the rate-distortion function, to explain the convergence of the EM and improved EM algorithms. This convergence proof adopts variational and iterative methods that Shannon et al. used for analyzing rate-distortion functions. The initialization map can vastly save both algorithms' running times for binary Gaussian mixtures. The FCP and the initialization map are useful for complicated mixtures but not sufficient; we need further studies for specific methods.