Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development

TL;DR

This paper introduces a realistic benchmark and evaluation framework for LLM-based autonomous agents in end-to-end software development, revealing key challenges and guiding future improvements.

Contribution

It presents a new challenging benchmark and a hybrid evaluation framework for assessing LLM-based agents, along with an empirical study on agent architecture impacts.

Findings

State-of-the-art agents fulfill about 50% of requirements.

Success depends heavily on task decomposition and collaboration strategies.

Main bottlenecks include requirement omission and poor self-verification.

Abstract

The development of LLM-based autonomous agents for end-to-end software development represents a significant paradigm shift in software engineering. However, the scientific evaluation of these systems is hampered by significant challenges, including overly simplistic benchmarks and the difficulty of conducting fair comparisons between different agent architectures due to confounding implementation variables. To address these limitations, we first construct a challenging and dynamically curated E2EDevBench to simulate realistic development scenarios. Second, we propose a hybrid evaluation framework that combines test-case-based functional assessment with fine-grained, LLM-based requirement verification. Using this framework, we conduct a controlled empirical study on three representative agent architectures implemented upon a unified foundation to isolate the impact of workflow design. Our findings reveal that state-of-the-art agents can fulfill approximately 50\% of requirements on \bench{}, but their success is critically dependent on the architectural strategy for task decomposition and collaboration. Furthermore, our analysis indicates that the primary bottleneck is the omission of requirements and inadequate self-verification. This work provides the community with a more realistic benchmark, a comprehensive evaluation framework, and crucial insights into the current capabilities and core challenges of software development agents, guiding future research toward enhancing requirement comprehension and planning.