Can Large Language Models Implement Agent-Based Models? An ODD-based Replication Study

TL;DR

This study evaluates whether large language models can reliably generate executable agent-based models from specifications, assessing their accuracy, efficiency, and scientific validity.

Contribution

It provides a systematic evaluation of 17 LLMs on a standardized agent-based modeling task, highlighting current capabilities and limitations.

Findings

GPT-4.1 produces valid, efficient implementations consistently.

Behaviorally faithful models are achievable but not guaranteed.

Executability alone is insufficient for scientific use.

Abstract

Large language models (LLMs) can now synthesize non-trivial executable code from textual descriptions, raising an important question: can LLMs reliably implement agent-based models from standardized specifications in a way that supports replication, verification, and validation? We address this question by evaluating 17 contemporary LLMs on a controlled ODD-to-code translation task, using the PPHPC predator-prey model as a fully specified reference. Generated Python implementations are assessed through staged executability checks, model-independent statistical comparison against a validated NetLogo baseline, and quantitative measures of runtime efficiency and maintainability. Results show that behaviorally faithful implementations are achievable but not guaranteed, and that executability alone is insufficient for scientific use. GPT-4.1 consistently produces statistically valid and efficient implementations, with Claude 3.7 Sonnet performing well but less reliably. Overall, the findings clarify both the promise and current limitations of LLMs as model engineering tools, with implications for reproducible agent-based and ecological modeling.