A Dynamical Systems Approach for Convergence of the Bayesian EM Algorithm

TL;DR

This paper applies Lyapunov stability theory from systems and control to analyze and establish convergence conditions for the Bayesian EM algorithm in machine learning, revealing potential for faster convergence under certain assumptions.

Contribution

It introduces a dynamical systems framework using Lyapunov stability to analyze convergence of the MAP-EM algorithm, extending the theoretical understanding of its behavior.

Findings

Provided convergence conditions for MAP-EM using Lyapunov theory

Shown that under certain assumptions, EM can achieve linear or quadratic convergence

Expanded the set of sufficient conditions for EM algorithm applications

Abstract

Out of the recent advances in systems and control (S\&C)-based analysis of optimization algorithms, not enough work has been specifically dedicated to machine learning (ML) algorithms and its applications. This paper addresses this gap by illustrating how (discrete-time) Lyapunov stability theory can serve as a powerful tool to aid, or even lead, in the analysis (and potential design) of optimization algorithms that are not necessarily gradient-based. The particular ML problem that this paper focuses on is that of parameter estimation in an incomplete-data Bayesian framework via the popular optimization algorithm known as maximum a posteriori expectation-maximization (MAP-EM). Following first principles from dynamical systems stability theory, conditions for convergence of MAP-EM are developed. Furthermore, if additional assumptions are met, we show that fast convergence (linear or quadratic) is achieved, which could have been difficult to unveil without our adopted S\&C approach. The convergence guarantees in this paper effectively expand the set of sufficient conditions for EM applications, thereby demonstrating the potential of similar S\&C-based convergence analysis of other ML algorithms.