Develop AI Agents for System Engineering in Factorio

TL;DR

This paper advocates for using the game Factorio as a sandbox environment to train and evaluate AI agents' system engineering skills, emphasizing dynamic system management and long-term planning.

Contribution

It proposes a new benchmark approach using automation-oriented sandbox games to better assess AI agents' system engineering capabilities.

Findings

Highlights limitations of current static benchmarks.

Suggests sandbox games as effective training environments.

Encourages research on AI reasoning and planning in engineering contexts.

Abstract

Continuing advances in frontier model research are paving the way for widespread deployment of AI agents. Meanwhile, global interest in building large, complex systems in software, manufacturing, energy and logistics has never been greater. Although AI driven system engineering holds tremendous promise, the static benchmarks dominating agent evaluations today fail to capture the crucial skills required for implementing dynamic systems, such as managing uncertain trade-offs and ensuring proactive adaptability. This position paper advocates for training and evaluating AI agents' system engineering abilities through automation-oriented sandbox games-particularly Factorio. By directing research efforts in this direction, we can equip AI agents with the specialized reasoning and long-horizon planning necessary to design, maintain, and optimize tomorrow's most demanding engineering projects.