Expressivity of Hidden Markov Chains vs. Recurrent Neural Networks from a system theoretic viewpoint

TL;DR

This paper compares the expressivity of Hidden Markov Chains and Recurrent Neural Networks by embedding them in a unified model and analyzing their covariance structures using stochastic realization theory.

Contribution

It introduces a unified generative model for HMC and RNN and provides a theoretical framework to compare their expressivity through structured covariance series.

Findings

Conditions for realizability of covariance series by HMC, RNN, or GUM

Comparison of model expressivity via structured covariance analysis

Theoretical insights into the generative capabilities of HMC and RNN

Abstract

Hidden Markov Chains (HMC) and Recurrent Neural Networks (RNN) are two well known tools for predicting time series. Even though these solutions were developed independently in distinct communities, they share some similarities when considered as probabilistic structures. So in this paper we first consider HMC and RNN as generative models, and we embed both structures in a common generative unified model (GUM). We next address a comparative study of the expressivity of these models. To that end we assume that the models are furthermore linear and Gaussian. The probability distributions produced by these models are characterized by structured covariance series, and as a consequence expressivity reduces to comparing sets of structured covariance series, which enables us to call for stochastic realization theory (SRT). We finally provide conditions under which a given covariance series can be realized by a GUM, an HMC or an RNN.