On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors

TL;DR

This paper investigates how overparametrization and priors influence Bayesian neural network posteriors, revealing phenomena that reshape their geometry and lead to structured, prior-aligned weight distributions.

Contribution

It provides a theoretical and empirical analysis of the interplay between overparametrization and priors in BNNs, highlighting key phenomena that alter posterior structure.

Findings

Redundancy causes balancedness, weight reallocation, and prior conformity.

Overparametrization leads to structured, prior-aligned posteriors.

Extensive experiments validate the theoretical insights.

Abstract

Bayesian neural network (BNN) posteriors are often considered impractical for inference, as symmetries fragment them, non-identifiabilities inflate dimensionality, and weight-space priors are seen as meaningless. In this work, we study how overparametrization and priors together reshape BNN posteriors and derive implications allowing us to better understand their interplay. We show that redundancy introduces three key phenomena that fundamentally reshape the posterior geometry: balancedness, weight reallocation on equal-probability manifolds, and prior conformity. We validate our findings through extensive experiments with posterior sampling budgets that far exceed those of earlier works, and demonstrate how overparametrization induces structured, prior-aligned weight posterior distributions.