Asymptotics of Continuous Bayes for Non-i.i.d. Sources

TL;DR

This paper extends the analysis of Bayesian methods' asymptotic performance from i.i.d. sources to more general non-stationary and dependent sources, showing they remain effective for compression and prediction.

Contribution

It provides a theoretical framework demonstrating Bayesian methods' effectiveness for non-i.i.d. sources, generalizing previous results to dependent and non-stationary cases.

Findings

Bayesian methods perform well for non-i.i.d. sources

Asymptotic relative entropy bounds are established

Results hold under mild technical assumptions

Abstract

Clarke and Barron analysed the relative entropy between an i.i.d. source and a Bayesian mixture over a continuous class containing that source. In this paper a comparable result is obtained when the source is permitted to be both non-stationary and dependent. The main theorem shows that Bayesian methods perform well for both compression and sequence prediction even in this most general setting with only mild technical assumptions.