The Transferability Limits of Static Benchmarks

TL;DR

This paper critically examines the limitations of static benchmarking in quantum chemistry, highlighting the uncertainty in error transferability and proposing a dynamic, system-focused approach for more reliable error quantification.

Contribution

It reveals the shortcomings of traditional static benchmarks and introduces a rolling, system-specific method to better quantify uncertainties in quantum chemical calculations.

Findings

Static benchmarks have limited transferability for error estimation.

Statistical analysis shows uncertainty depends on reference data choice.

A system-focused, dynamic benchmarking approach improves reliability.

Abstract

Every practical method to solve the Schr\"odinger equation for interacting many-particle systems introduces approximations. Such methods are therefore plagued by systematic errors. For computational chemistry, it is decisive to quantify the specific error for some system under consideration. Traditionally, the primary resource for such an error assessment have been benchmarking results, usually taken from the literature. However, their transferability to a specific molecular system, and hence, the reliability of the traditional approach always remains uncertain to some degree. In this communication, we elaborate on the shortcomings of this traditional way of static benchmarking by exploiting statistical analyses at the example of one of the largest quantum chemical benchmark sets available. We demonstrate the uncertainty of error estimates in the light of the choice of reference data selected for a benchmark study. To alleviate the issues with static benchmarks, we advocate to rely instead on a rolling and system-focused approach for rigorously quantifying the uncertainty of a quantum chemical result.