Consistent Entropy Estimation for Stationary Time Series

TL;DR

This paper provides rigorous bias bounds for entropy estimators based on k-nearest neighbors applied to stationary time series, extending their theoretical understanding beyond independent data.

Contribution

It establishes bias decay rates for K-nearest neighbor entropy estimators under stationary mixing conditions, with numerical validation on Gaussian Markov processes.

Findings

Bias decays at a quantifiable rate with sample size N

Estimator performs efficiently on stationary Gaussian Markov processes

Theoretical bounds are supported by numerical experiments

Abstract

Entropy estimation, due in part to its connection with mutual information, has seen considerable use in the study of time series data including causality detection and information flow. In many cases, the entropy is estimated using $k$-nearest neighbor (Kozachenko-Leonenko) based methods. However, analytic results on this estimator are limited to independent data. In the article, we show rigorous bounds on the rate of decay of the bias in the number of samples, $N$, assuming they are drawn from a stationary process which satisfies a suitable mixing condition. Numerical examples are presented which demonstrate the efficiency of the estimator when applied to a Markov process with stationary Gaussian density. These results support the asymptotic rates derived in the theoretical work.