Parametrization invariant interpretation of priors and posteriors

TL;DR

This paper introduces a Riemannian manifold approach to interpret priors and posteriors in Bayesian inference, emphasizing invariance to parameterization by considering distributions over probability distributions.

Contribution

It proposes a parametrization-invariant framework for Bayesian priors and posteriors using Riemannian geometry, shifting the perspective from parameters to distributions over distributions.

Findings

Invariance to parametrization achieved for Bayesian estimates.

Analysis of Bernoulli distributions exemplifies the approach.

Maximum a posteriori estimates become parametrization-independent.

Abstract

In this paper we leverage on probability over Riemannian manifolds to rethink the interpretation of priors and posteriors in Bayesian inference. The main mindshift is to move away from the idea that "a prior distribution establishes a probability distribution over the parameters of our model" to the idea that "a prior distribution establishes a probability distribution over probability distributions". To do that we assume that our probabilistic model is a Riemannian manifold with the Fisher metric. Under this mindset, any distribution over probability distributions should be "intrinsic", that is, invariant to the specific parametrization which is selected for the manifold. We exemplify our ideas through a simple analysis of distributions over the manifold of Bernoulli distributions. One of the major shortcomings of maximum a posteriori estimates is that they depend on the parametrization. Based on the understanding developed here, we can define the maximum a posteriori estimate which is independent of the parametrization.