Geometry of EM and related iterative algorithms

TL;DR

This paper introduces an information geometric perspective on the EM algorithm, extending its applications to robust inference, channel capacity, PCA, modal regression, matrix factorization, and deep generative models.

Contribution

It presents a unified geometric framework for EM and related algorithms, enabling new applications in robust inference and deep learning.

Findings

Geometric formulation of EM unifies various algorithms.

Extension to outlier-robust inference and channel capacity.

Application to deep generative models and multivariate analysis.

Abstract

The Expectation--Maximization (EM) algorithm is a simple meta-algorithm that has been used for many years as a methodology for statistical inference when there are missing measurements in the observed data or when the data is composed of observables and unobservables. Its general properties are well studied, and also, there are countless ways to apply it to individual problems. In this paper, we introduce the $em$ algorithm, an information geometric formulation of the EM algorithm, and its extensions and applications to various problems. Specifically, we will see that it is possible to formulate an outlier-robust inference algorithm, an algorithm for calculating channel capacity, parameter estimation methods on probability simplex, particular multivariate analysis methods such as principal component analysis in a space of probability models and modal regression, matrix factorization, and learning generative models, which have recently attracted attention in deep learning, from the geometric perspective.