Bayesian Effective Dimension: A Mutual Information Perspective

TL;DR

This paper introduces the Bayesian effective dimension, a mutual information-based measure that quantifies how many directions in parameter space are statistically learnable, explaining dimension reduction in high-dimensional Bayesian models.

Contribution

It defines the Bayesian effective dimension, linking mutual information with model complexity, and demonstrates its properties and calculations in Gaussian and linear models.

Findings

Effective dimension equals parameter dimension in regular models.

Shrinkage and regularization reduce effective dimension.

Explicit calculations connect effective dimension with spectral properties.

Abstract

High-dimensional Bayesian procedures often exhibit behavior that is effectively low dimensional, even when the ambient parameter space is large or infinite-dimensional. This phenomenon underlies the success of shrinkage priors, regularization, and approximate Bayesian methods, yet it is typically described only informally through notions such as sparsity, intrinsic dimension, or degrees of freedom. In this paper we introduce the \emph{Bayesian effective dimension}, a model- and prior-dependent quantity defined through the mutual information between parameters and data. This notion quantifies the expected information gain from prior to posterior and provides a coordinate-free measure of how many directions in parameter space are statistically learnable at a given sample size. In regular parametric models the effective dimension coincides with the usual parameter dimension, while in high-dimensional, ill-posed, or strongly regularized settings it can be substantially smaller. We develop basic properties of the effective dimension and present explicit calculations for Gaussian location models and linear models with general design, revealing close connections with spectral complexity and effective rank. These examples illustrate how shrinkage and regularization mechanisms directly control the growth of effective dimension. The framework offers a unifying perspective on dimension reduction in Bayesian inference and provides insight into uncertainty quantification and the behavior of approximate posterior distributions.