Truly Bayesian Entropy Estimation

TL;DR

This paper introduces a fully Bayesian method for estimating the entropy rate of discrete time series using Bayesian Context Trees, providing comprehensive posterior distributions and outperforming existing methods on various datasets.

Contribution

It presents a novel Bayesian approach for entropy estimation based on Bayesian Context Trees, enabling direct sampling from the posterior and offering richer insights than traditional point estimates.

Findings

The method achieves consistency and asymptotic normality.

It outperforms state-of-the-art alternatives on simulated data.

Practical utility demonstrated on real-world datasets.

Abstract

Estimating the entropy rate of discrete time series is a challenging problem with important applications in numerous areas including neuroscience, genomics, image processing and natural language processing. A number of approaches have been developed for this task, typically based either on universal data compression algorithms, or on statistical estimators of the underlying process distribution. In this work, we propose a fully-Bayesian approach for entropy estimation. Building on the recently introduced Bayesian Context Trees (BCT) framework for modelling discrete time series as variable-memory Markov chains, we show that it is possible to sample directly from the induced posterior on the entropy rate. This can be used to estimate the entire posterior distribution, providing much richer information than point estimates. We develop theoretical results for the posterior distribution of the entropy rate, including proofs of consistency and asymptotic normality. The practical utility of the method is illustrated on both simulated and real-world data, where it is found to outperform state-of-the-art alternatives.