GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Libraries

TL;DR

This paper benchmarks various large language models, especially GPT-4.1, for automated Python code generation in scientific experiments, demonstrating GPT-4.1's superior performance and highlighting current limitations in LLM-based scientific automation.

Contribution

It provides a systematic evaluation of LLMs' ability to generate functional Python code for complex scientific tasks, introducing novel benchmarking methods and revealing GPT-4.1's exceptional success.

Findings

GPT-4.1 achieved 100% success rate in code generation tasks.

Most models succeeded in fewer than half of the runs.

Identified shortcomings in third-party libraries affecting code execution.

Abstract

Large Language Models (LLMs) have advanced rapidly as tools for automating code generation in scientific research, yet their ability to interpret and use unfamiliar Python APIs for complex computational experiments remains poorly characterized. This study systematically benchmarks a selection of state-of-the-art LLMs in generating functional Python code for two increasingly challenging scenarios: conversational data analysis with the \textit{ParShift} library, and synthetic data generation and clustering using \textit{pyclugen} and \textit{scikit-learn}. Both experiments use structured, zero-shot prompts specifying detailed requirements but omitting in-context examples. Model outputs are evaluated quantitatively for functional correctness and prompt compliance over multiple runs, and qualitatively by analyzing the errors produced when code execution fails. Results show that only a small subset of models consistently generate correct, executable code. GPT-4.1 achieved a 100\% success rate across all runs in both experimental tasks, whereas most other models succeeded in fewer than half of the runs, with only Grok-3 and Mistral-Large approaching comparable performance. In addition to benchmarking LLM performance, this approach helps identify shortcomings in third-party libraries, such as unclear documentation or obscure implementation bugs. Overall, these findings highlight current limitations of LLMs for end-to-end scientific automation and emphasize the need for careful prompt design, comprehensive library documentation, and continued advances in language model capabilities.