Posterior Representations for Bayesian Context Trees: Sampling, Estimation and Convergence

TL;DR

This paper introduces a new branching process representation for Bayesian Context Trees, enabling efficient sampling, establishing posterior consistency, and demonstrating superior entropy estimation in discrete time series analysis.

Contribution

It presents a novel branching process approach for Bayesian Context Trees, improving sampling efficiency and providing theoretical guarantees for posterior convergence.

Findings

The new sampler outperforms previous MCMC methods in efficiency.

The posterior distribution is shown to be asymptotically consistent.

The Bayesian entropy estimator surpasses existing methods in experiments.

Abstract

We revisit the Bayesian Context Trees (BCT) modelling framework for discrete time series, which was recently found to be very effective in numerous tasks including model selection, estimation and prediction. A novel representation of the induced posterior distribution on model space is derived in terms of a simple branching process, and several consequences of this are explored in theory and in practice. First, it is shown that the branching process representation leads to a simple variable-dimensional Monte Carlo sampler for the joint posterior distribution on models and parameters, which can efficiently produce independent samples. This sampler is found to be more efficient than earlier MCMC samplers for the same tasks. Then, the branching process representation is used to establish the asymptotic consistency of the BCT posterior, including the derivation of an almost-sure convergence rate. Finally, an extensive study is carried out on the performance of the induced Bayesian entropy estimator. Its utility is illustrated through both simulation experiments and real-world applications, where it is found to outperform several state-of-the-art methods.