Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference

TL;DR

This paper introduces Gibbs priors as a diagnostic tool to identify and analyze the inductive bias introduced by approximate Bayesian inference methods, providing a practical way to assess inference reliability.

Contribution

It proposes a novel diagnostic approach using Gibbs priors to reverse-engineer the inductive bias in approximate Bayesian inference, applicable across various models and approximations.

Findings

Gibbs priors effectively reveal inductive biases in Gaussian models.

The diagnostic is easy to implement with pseudo-Gibbs sampling.

Applicable to diverse Bayesian models and approximation techniques.

Abstract

Full Bayesian posteriors are rarely analytically tractable, which is why real-world Bayesian inference heavily relies on approximate techniques. Approximations generally differ from the true posterior and require diagnostic tools to assess whether the inference can still be trusted. We investigate a new approach to diagnosing approximate inference: the approximation mismatch is attributed to a change in the inductive bias by treating the approximations as exact and reverse-engineering the corresponding prior. We show that the problem is more complicated than it appears to be at first glance, because the solution generally depends on the observation. By reframing the problem in terms of incompatible conditional distributions we arrive at a natural solution: the Gibbs prior. The resulting diagnostic is based on pseudo-Gibbs sampling, which is widely applicable and easy to implement. We illustrate how the Gibbs prior can be used to discover the inductive bias in a controlled Gaussian setting and for a variety of Bayesian models and approximations.