Tests for model misspecification in simulation-based inference: from local distortions to global model checks

TL;DR

This paper develops a comprehensive framework for detecting model misspecification in simulation-based inference, connecting classical techniques with modern hypothesis testing, and demonstrates its effectiveness on gravitational wave data.

Contribution

It introduces a flexible, hypothesis-testing-based framework for model discrepancy analysis in SBI, including an efficient self-calibrating training algorithm and real-world application.

Findings

Framework performs well across multiple scenarios

Connections established with classical anomaly detection and validation methods

Successfully applied to gravitational wave event GW150914

Abstract

Model misspecification analysis strategies, such as anomaly detection, model validation, and model comparison are a key component of scientific model development. Over the last few years, there has been a rapid rise in the use of simulation-based inference (SBI) techniques for Bayesian parameter estimation, applied to increasingly complex forward models. To move towards fully simulation-based analysis pipelines, however, there is an urgent need for a comprehensive simulation-based framework for model misspecification analysis. In this work, we provide a solid and flexible foundation for a wide range of model discrepancy analysis tasks, using distortion-driven model misspecification tests. From a theoretical perspective, we introduce the statistical framework built around performing many hypothesis tests for distortions of the simulation model. We also make explicit analytic connections to classical techniques: anomaly detection, model validation, and goodness-of-fit residual analysis. Furthermore, we introduce an efficient self-calibrating training algorithm that is useful for practitioners. We demonstrate the performance of the framework in multiple scenarios, making the connection to classical results where they are valid. Finally, we show how to conduct such a distortion-driven model misspecification test for real gravitational wave data, specifically on the event GW150914.