EnvTrace: Simulation-Based Semantic Evaluation of LLM Code via Execution Trace Alignment -- Demonstrated at Synchrotron Beamlines

TL;DR

EnvTrace introduces a simulation-based evaluation method for LLM-generated control code using execution trace alignment, demonstrated at synchrotron beamlines, enabling more accurate assessment of models' semantic understanding in physical system control.

Contribution

The paper presents EnvTrace, a novel simulation-based evaluation framework that assesses LLMs' control code through execution trace alignment, bridging the gap between language models and physical system validation.

Findings

Many top-tier LLMs approach human-level performance in control code generation.

EnvTrace effectively evaluates functional correctness across multiple behavioral dimensions.

Digital twins enable pre-execution validation and safe testing of control code.

Abstract

Evaluating large language models (LLMs) for instrument control requires methods that go beyond standard, stateless algorithmic benchmarks, since the behavior of physical systems cannot be fully captured by unit tests alone. Here we introduce EnvTrace, a simulation-based method that evaluates execution traces to assess semantic code equivalence. EnvTrace is demonstrated with a beamline control-logic digital twin to facilitate the evaluation of instrument control code, with the digital twin itself also enabling the pre-execution validation of live experiments. Over 30 LLMs were evaluated using trace alignment to generate a multi-faceted score for functional correctness across key behavioral dimensions, showing that many top-tier models can approach human-level performance in rapid control-code generation. This is a first step toward a broader vision where LLMs and digital twins work symbiotically: LLMs providing intuitive control and agentic orchestration, and digital twins offering safe and high-fidelity environments, paving the way towards autonomous embodied AI.