ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas

TL;DR

ProxyWar introduces a dynamic, game-based framework for evaluating LLM-generated code by embedding agents in competitive environments, revealing limitations of static benchmarks and guiding future improvements.

Contribution

It presents a novel, holistic evaluation method combining automated testing, iterative repair, and multi-agent tournaments for assessing LLM code in dynamic settings.

Findings

Benchmark scores often do not reflect real-world performance.

Dynamic evaluation uncovers limitations of current LLM code generators.

The framework enables research into adaptive problem solving and algorithm discovery.

Abstract

Large language models (LLMs) have revolutionized automated code generation, yet the evaluation of their real-world effectiveness remains limited by static benchmarks and simplistic metrics. We present ProxyWar, a novel framework that systematically assesses code generation quality by embedding LLM-generated agents within diverse, competitive game environments. Unlike existing approaches, ProxyWar evaluates not only functional correctness but also the operational characteristics of generated programs, combining automated testing, iterative code repair, and multi-agent tournaments to provide a holistic view of program behavior. Applied to a range of state-of-the-art coders and games, our approach uncovers notable discrepancies between benchmark scores and actual performance in dynamic settings, revealing overlooked limitations and opportunities for improvement. These findings highlight the need for richer, competition-based evaluation of code generation. Looking forward, ProxyWar lays a foundation for research into LLM-driven algorithm discovery, adaptive problem solving, and the study of practical efficiency and robustness, including the potential for models to outperform hand-crafted agents. The project is available at https://github.com/xinke-wang/ProxyWar.