Redundancy of Exchangeable Estimators

TL;DR

This paper investigates the redundancy of Bayesian estimators derived from Poisson-Dirichlet and Pitman-Yor priors, linking data compression and statistical inference in large alphabet settings to identify regimes of low redundancy.

Contribution

It provides a novel analysis of the redundancy of exchangeable estimators in the context of universal data compression and Bayesian inference for large discrete alphabets.

Findings

Redundancy is small when sample size and alphabet size are appropriately related.

Identifies regimes where Bayesian estimators are efficient for large alphabet data.

Connects Bayesian nonparametric priors with universal compression performance.

Abstract

Exchangeable random partition processes are the basis for Bayesian approaches to statistical inference in large alphabet settings. On the other hand, the notion of the pattern of a sequence provides an information-theoretic framework for data compression in large alphabet scenarios. Because data compression and parameter estimation are intimately related, we study the redundancy of Bayes estimators coming from Poisson-Dirichlet priors (or "Chinese restaurant processes") and the Pitman-Yor prior. This provides an understanding of these estimators in the setting of unknown discrete alphabets from the perspective of universal compression. In particular, we identify relations between alphabet sizes and sample sizes where the redundancy is small, thereby characterizing useful regimes for these estimators.