MIRA: A Score for Conditional Distribution Accuracy and Model Comparison

TL;DR

Mira is a novel sample-based score for evaluating and comparing the accuracy of conditional distributions using joint samples, facilitating Bayesian model validation without complex evidence calculations.

Contribution

The paper introduces Mira, a new analytic score for assessing conditional distribution accuracy and enabling model comparison directly from joint samples.

Findings

Mira accurately assesses conditional distribution alignment in toy and Bayesian inference problems.

It allows Bayesian model comparison without computing evidence.

The Mira score provides theoretical reference values and uncertainty estimates.

Abstract

We introduce Mira, a sample-based score for assessing the accuracy of a candidate conditional distribution using only joint samples from the true data-generating process. Relying on the principle that distributions coincide if they assign equal probability mass to all regions, we derive an analytic expression for the Mira statistic, whose average defines the Mira score. This formulation further allows us to compute theoretical reference values and uncertainty estimates when the candidate distribution matches the true one. This framework enables model comparison by quantifying the alignment between the conditional distribution of a candidate model and the true data generating process. Consequently, Mira enables Bayesian model comparison through direct posterior validation, bypassing the challenging evidence computation. We demonstrate its effectiveness across several toy problems and Bayesian inference tasks.