EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems

TL;DR

EvoCodeBench is a new benchmark designed to evaluate self-evolving large language model-driven coding systems, measuring their iterative improvement, efficiency, and robustness across languages compared to human programmers.

Contribution

It introduces a comprehensive benchmark that captures inference-time self-evolution, resource costs, and cross-language stability, with direct comparison to human performance.

Findings

Self-evolving systems show measurable efficiency gains over time.

Benchmark enables analysis of cross-language robustness and long-tail language stability.

Human performance provides a valuable reference point for evaluating LLM coding systems.

Abstract

As large language models (LLMs) continue to advance in programming tasks, LLM-driven coding systems have evolved from one-shot code generation into complex systems capable of iterative improvement during inference. However, existing code benchmarks primarily emphasize static correctness and implicitly assume fixed model capability during inference. As a result, they do not capture inference-time self-evolution, such as whether accuracy and efficiency improve as an agent iteratively refines its solutions. They also provide limited accounting of resource costs and rarely calibrate model performance against that of human programmers. Moreover, many benchmarks are dominated by high-resource languages, leaving cross-language robustness and long-tail language stability underexplored. Therefore, we present EvoCodeBench, a benchmark for evaluating self-evolving LLM-driven coding systems across programming languages with direct comparison to human performance. EvoCodeBench tracks performance dynamics, measuring solution correctness alongside efficiency metrics such as solving time, memory consumption, and improvement algorithmic design over repeated problem-solving attempts. To ground evaluation in a human-centered reference frame, we directly compare model performance with that of human programmers on the same tasks, enabling relative performance assessment within the human ability distribution. Furthermore, EvoCodeBench supports multiple programming languages, enabling systematic cross-language and long-tail stability analyses under a unified protocol. Our results demonstrate that self-evolving systems exhibit measurable gains in efficiency over time, and that human-relative and multi-language analyses provide insights unavailable through accuracy alone. EvoCodeBench establishes a foundation for evaluating coding intelligence in evolving LLM-driven systems.