All Emulators are Wrong, Many are Useful, and Some are More Useful Than Others: A Reproducible Comparison of Computer Model Surrogates

TL;DR

This paper provides a comprehensive, reproducible comparison of 29 surrogate modeling methods across diverse datasets, introducing the uqlingmulation framework to facilitate standardized benchmarking and guide practitioners.

Contribution

It introduces the uqlingramework for reproducible emulator benchmarking and offers an empirical comparison of 29 methods across multiple datasets.

Findings

Empirical insights into strengths and weaknesses of state-of-the-art emulators.

Guidance for selecting surrogates based on dataset characteristics.

Framework enables reproducible and extendable emulator comparisons.

Abstract

Accurate and efficient surrogate modeling is essential for modern computational science, and there are a staggering number of emulation methods to choose from. With new methods being developed all the time, comparing the relative strengths and weaknesses of different methods remains a challenge due to inconsistent benchmarking practices and (sometimes) limited reproducibility and transparency. In this work, we present a large-scale, fully reproducible comparison of $29$ distinct emulators across $60$ canonical test functions and $40$ real emulation datasets. To facilitate rigorous, apples-to-apples comparisons, we introduce the R package \texttt{duqling}, which streamlines reproducible simulation studies using a consistent, simple syntax, and automatic internal scaling of inputs. This framework allows researchers to compare emulators in a unified environment and makes it possible to replicate or extend previous studies with minimal effort, even across different publications. Our results provide detailed empirical insight into the strengths and weaknesses of state-of-the-art emulators and offer guidance for both method developers and practitioners selecting a surrogate for new data. We discuss best practices for emulator comparison and highlight how \texttt{duqling} can accelerate research in emulator design and application.