On a divergence-based prior analysis of stick-breaking processes

TL;DR

This paper analyzes the distributional differences among various nonparametric priors, especially stick-breaking processes, using divergence measures, and provides conditions under which simpler priors are preferable.

Contribution

It introduces a divergence-based framework to compare nonparametric priors, extending analysis to a broad class of exchangeable processes, and offers quantitative criteria for prior selection.

Findings

Computed mean and variance of Kullback-Leibler divergence between Dirichlet and geometric processes.

Extended analysis to a broad class of exchangeable stick-breaking processes.

Established conditions where priors are close in total variation, favoring simpler models.

Abstract

The nonparametric view of Bayesian inference has transformed statistics and many of its applications. The canonical Dirichlet process and other more general families of nonparametric priors have served as a gateway to solve frontier uncertainty quantification problems of large, or infinite, nature. This success has been greatly due to available constructions and representations of such distributions, the two most useful constructions are the one based on normalization of homogeneous completely random measures and that based on stick-breaking processes. Hence, understanding their distributional features and how different random probability measures compare among themselves is a key ingredient for their proper application. In this paper, we analyse the discrepancy among some nonparametric priors employed in the literature. Initially, we compute the mean and variance of the random Kullback-Leibler divergence between the Dirichlet process and the geometric process. Subsequently, we extend our analysis to encompass a broader class of exchangeable stick-breaking processes, which includes the Dirichlet and geometric processes as extreme cases. Our results establish quantitative conditions where all the aforementioned priors are close in total variation distance. In such instances, adhering to Occam's razor principle advocates for the preference of the simpler process.