A Tutorial on the Expectation-Maximization Algorithm Including Maximum-Likelihood Estimation and EM Training of Probabilistic Context-Free Grammars

TL;DR

This paper provides a comprehensive tutorial on the Expectation-Maximization algorithm, covering its theoretical foundations, estimation methods, and applications to probabilistic context-free grammars, with illustrative examples.

Contribution

It offers a clear, mathematically grounded explanation of EM and demonstrates its application to probabilistic context-free grammars, enhancing understanding for researchers and practitioners.

Findings

EM algorithm effectively estimates probabilistic grammars

Comparison of relative-frequency and maximum-likelihood estimation methods

Illustrative examples demonstrate practical application

Abstract

The paper gives a brief review of the expectation-maximization algorithm (Dempster 1977) in the comprehensible framework of discrete mathematics. In Section 2, two prominent estimation methods, the relative-frequency estimation and the maximum-likelihood estimation are presented. Section 3 is dedicated to the expectation-maximization algorithm and a simpler variant, the generalized expectation-maximization algorithm. In Section 4, two loaded dice are rolled. A more interesting example is presented in Section 5: The estimation of probabilistic context-free grammars.