Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPS

TL;DR

This paper introduces an evaluation method for assessing the quality of LLM-generated input files for LAMMPS, a molecular dynamics software, highlighting current limitations and proposing a practical integration approach for domain experts.

Contribution

It presents a novel evaluation procedure for LLM-generated scientific code in domain-specific languages, enabling expert assessment without extensive testing.

Findings

LLMs have limitations in generating valid scientific DSL code.

The evaluation procedure effectively identifies common errors in LLM outputs.

The approach facilitates integration of LLMs into scientific computational workflows.

Abstract

Large language models (LLMs) are changing the way researchers interact with code and data in scientific computing. While their ability to generate general-purpose code is well established, their effectiveness in producing scientifically valid code/input scripting for domain-specific languages (DSLs) remains largely unexplored. We propose an evaluation procedure that enables domain experts (who may not be experts in the DSL) to assess the validity of LLM-generated input files for LAMMPS, a widely used molecular dynamics (MD) code, and to use those assessments to evaluate the performance of state-of-the-art LLMs and identify common issues. Key to the evaluation procedure are a normalization step to generate canonical files and an extensible parser for syntax analysis. The following steps isolate common errors without incurring costly tests (in time and computational resources). Once a working input file is generated, LLMs can accelerate verification tests. Our findings highlight limitations of LLMs in generating scientific DSLs and a practical path forward for their integration into domain-specific computational ecosystems by domain experts.