Benchmarking Self-Driving Labs

TL;DR

This paper reviews the progress and metrics in benchmarking self-driving labs, highlighting their ability to accelerate materials discovery by reducing experiments needed and analyzing factors influencing their efficiency.

Contribution

It introduces and discusses key metrics like AF and EF, and provides a comprehensive review of literature and simulations to understand SDL performance.

Findings

Median acceleration factor (AF) is 6, increasing with dimensionality.

EF varies widely but peaks at 10-20 experiments per dimension.

EF depends on parameter space properties, AF on complexity.

Abstract

A key goal of modern materials science is accelerating the pace of materials discovery. Self-driving labs, or systems that select experiments using machine learning and then execute them using automation, are designed to fulfil this promise by performing experiments faster, more intelligently, more reliably, and with richer metadata than conventional means. This review summarizes progress in understanding the degree to which SDLs accelerate learning by quantifying how much they reduce the number of experiments required for a given goal. The review begins by summarizing the theory underlying two key metrics, namely acceleration factor AF and enhancement factor EF, which quantify how much faster and better an algorithm is relative to a reference strategy. Next, we provide a comprehensive review of the literature, which reveals a wide range of AFs with a median of 6, and that tends to increase with the dimensionality of the space, reflecting an interesting blessing of dimensionality. In contrast, reported EF values vary by over two orders of magnitude, although they consistently peak at 10-20 experiments per dimension. To understand these results, we perform a series of simulated Bayesian optimization campaigns that reveal how EF depends upon the statistical properties of the parameter space while AF depends on its complexity. Collectively, these results reinforce the motivation for using SDLs by revealing their value across a wide range of material parameter spaces and provide a common language for quantifying and understanding this acceleration.