Forward-reverse EM algorithm for Markov chains: convergence and numerical analysis

TL;DR

This paper introduces a forward-reverse EM algorithm for Markov chains, providing convergence proofs, complexity analysis, and demonstrating its applicability to various models including continuous and discrete time processes.

Contribution

The paper develops a novel forward-reverse EM algorithm for Markov chains with convergence proofs and complexity analysis, extending previous work on conditional diffusions.

Findings

Almost sure convergence of the FREM algorithm for curved exponential family Markov models

Complexity analysis deriving the expected computational cost

Application examples demonstrating versatility across different Markov models

Abstract

We develop a forward-reverse EM (FREM) algorithm for estimating parameters that determine the dynamics of a discrete time Markov chain evolving through a certain measurable state space. As a key tool for the construction of the FREM method we develop forward-reverse representations for Markov chains conditioned on a certain terminal state. These representations may be considered as an extension of the earlier work Bayer and Schoenmakers [2013] on conditional diffusions. We proof almost sure convergence of our algorithm for a Markov chain model with curved exponential family structure. On the numerical side we give a complexity analysis of the forward-reverse algorithm by deriving its expected cost. Two application examples are discuss to demonstrate the scope of possible applications ranging from models based on continuous time processes to discrete time Markov chain models.