Noise Benefits in Expectation-Maximization Algorithms

TL;DR

Injecting carefully calibrated noise into data can significantly accelerate Expectation-Maximization algorithms, demonstrating a form of stochastic resonance that benefits various statistical signal processing tasks.

Contribution

This work introduces conditions under which noise enhances EM algorithm convergence and demonstrates its effectiveness across multiple models and inference frameworks.

Findings

Noise injection speeds up EM convergence

Applicable to mixture models, k-means, HMMs, neural networks

Provides theoretical guarantees for Bayesian approximations

Abstract

This dissertation shows that careful injection of noise into sample data can substantially speed up Expectation-Maximization algorithms. Expectation-Maximization algorithms are a class of iterative algorithms for extracting maximum likelihood estimates from corrupted or incomplete data. The convergence speed-up is an example of a noise benefit or "stochastic resonance" in statistical signal processing. The dissertation presents derivations of sufficient conditions for such noise-benefits and demonstrates the speed-up in some ubiquitous signal-processing algorithms. These algorithms include parameter estimation for mixture models, the $k$-means clustering algorithm, the Baum-Welch algorithm for training hidden Markov models, and backpropagation for training feedforward artificial neural networks. This dissertation also analyses the effects of data and model corruption on the more general Bayesian inference estimation framework. The main finding is a theorem guaranteeing that uniform approximators for Bayesian model functions produce uniform approximators for the posterior pdf via Bayes theorem. This result also applies to hierarchical and multidimensional Bayesian models.