The entropy rate of Linear Additive Markov Processes

TL;DR

This paper introduces a theoretical framework for calculating the entropy rate of Linear Additive Markov Processes (LAMP), demonstrating that their entropy rate matches that of the underlying first-order Markov Chain, and shows LAMP's effectiveness in modeling complex data dependencies.

Contribution

The paper derives a theoretical entropy rate for LAMP, revealing its equivalence to the underlying Markov Chain's entropy rate, and applies it to real-world datasets for improved dependency modeling.

Findings

LAMP entropy rate matches the underlying Markov Chain's entropy rate.

LAMP models often yield lower entropy estimates than traditional methods.

LAMP effectively captures long-range dependencies in complex datasets.

Abstract

This work derives a theoretical value for the entropy of a Linear Additive Markov Process (LAMP), an expressive model able to generate sequences with a given autocorrelation structure. While a first-order Markov Chain model generates new values by conditioning on the current state, the LAMP model takes the transition state from the sequence's history according to some distribution which does not have to be bounded. The LAMP model captures complex relationships and long-range dependencies in data with similar expressibility to a higher-order Markov process. While a higher-order Markov process has a polynomial parameter space, a LAMP model is characterised only by a probability distribution and the transition matrix of an underlying first-order Markov Chain. We prove that the theoretical entropy rate of a LAMP is equivalent to the theoretical entropy rate of the underlying first-order Markov Chain. This surprising result is explained by the randomness introduced by the random process which selects the LAMP transitioning state, and provides a tool to model complex dependencies in data while retaining useful theoretical results. We use the LAMP model to estimate the entropy rate of the LastFM, BrightKite, Wikispeedia and Reuters-21578 datasets. We compare estimates calculated using frequency probability estimates, a first-order Markov model and the LAMP model, and consider two approaches to ensuring the transition matrix is irreducible. In most cases the LAMP entropy rates are lower than those of the alternatives, suggesting that LAMP model is better at accommodating structural dependencies in the processes.