Benchmarking Autonomy in Scientific Experiments: A Hierarchical Taxonomy for Autonomous Large-Scale Facilities

TL;DR

This paper introduces the BASE Scale, a six-level hierarchy for benchmarking autonomy in scientific experiments at large-scale facilities, addressing unique operational constraints and enabling standardized assessment of autonomous capabilities.

Contribution

It develops a novel hierarchical taxonomy tailored for autonomous scientific experiments, adapting existing standards to the specific needs of large-scale user facilities.

Findings

Defines the six levels of the BASE Scale for autonomy.

Identifies the Inference Barrier as a critical latency threshold.

Provides operational metrics for assessing autonomous experimental workflows.

Abstract

The transition from automated data collection to fully autonomous discovery requires a shared vocabulary to benchmark progress. While the automotive industry relies on the SAE J3016 standard, current taxonomies for autonomous science presuppose an owner-operator model that is incompatible with the operational rigidities of Large-Scale User Facilities. Here, we propose the Benchmarking Autonomy in Scientific Experiments (BASE) Scale, a 6-level taxonomy (Levels 0-5) specifically adapted for these unique constraints. Unlike owner-operator models, User Facilities require zero-shot deployment where agents must operate immediately without extensive training periods. We define the specific technical requirements for each tier, identifying the Inference Barrier (Level 3) as the critical latency threshold where decisions shift from scalar feedback to semantic digital twins. Fundamentally, this level extends the decision manifold from spatial exploration to temporal gating, enabling the agent to synchronise acquisition with the onset of transient physical events. By establishing these operational definitions, the BASE Scale provides facility directors, funding bodies, and beamline scientists with a standardised metric to assess risk, define liability, and quantify the intelligence of experimental workflows.