Simulation-based inference with deep ensembles: Evaluating calibration uncertainty and detecting model misspecification

TL;DR

This paper introduces an ensemble-based diagnostic method for Simulation-Based Inference (SBI) that assesses the consistency of posterior estimates, helping detect model misspecification and quantify uncertainty without needing the true posterior.

Contribution

The authors propose a simple, scalable ensemble learning diagnostic for SBI that evaluates internal consistency and detects model misspecification without access to the true posterior.

Findings

Ensemble KL divergence correlates with convergence issues.

The method detects model misspecification effectively.

Provides a scalable tool for SBI validation.

Abstract

Simulation-Based Inference (SBI) offers a principled and flexible framework for conducting Bayesian inference in any situation where forward simulations are feasible. However, validating the accuracy and reliability of the inferred posteriors remains a persistent challenge. In this work, we point out a simple diagnostic approach rooted in ensemble learning methods to assess the internal consistency of SBI outputs that does not require access to the true posterior. By training multiple neural estimators under identical conditions and evaluating their pairwise Kullback-Leibler (KL) divergences, we define a consistency criterion that quantifies agreement across the ensemble. We highlight two core use cases for this framework: a) for generating a robust estimate of the systematic uncertainty in parameter reconstruction associated with the training procedure, and b) for detecting possible model misspecification when using trained estimators on real data. We also demonstrate the relationship between significant KL divergences and issues such as insufficient convergence due to, e.g., too low a simulation budget, or intrinsic variance in the training process. Overall, this ensemble-based diagnostic framework provides a lightweight, scalable, and model-agnostic tool for enhancing the trustworthiness of SBI in scientific applications.