Benchmarking Simulation-Based Inference

TL;DR

This paper introduces a comprehensive benchmark for simulation-based inference algorithms, highlighting the importance of performance metrics, comparing various methods, and providing insights to improve likelihood-free inference techniques.

Contribution

It provides the first public benchmark with tasks and metrics for likelihood-free inference, including an initial comparison of recent neural network and classical methods.

Findings

Performance metric choice is critical

Sequential estimation improves sample efficiency

Neural network approaches generally outperform classical methods

Abstract

Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. However, a public benchmark with appropriate performance metrics for such 'likelihood-free' algorithms has been lacking. This has made it difficult to compare algorithms and identify their strengths and weaknesses. We set out to fill this gap: We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms including recent approaches employing neural networks and classical Approximate Bayesian Computation methods. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency. Neural network-based approaches generally exhibit better performance, but there is no uniformly best algorithm. We provide practical advice and highlight the potential of the benchmark to diagnose problems and improve algorithms. The results can be explored interactively on a companion website. All code is open source, making it possible to contribute further benchmark tasks and inference algorithms.