A Comprehensive Empirical Evaluation of Agent Frameworks on Code-centric Software Engineering Tasks

TL;DR

This paper provides a comprehensive empirical evaluation of seven agent frameworks across key software engineering tasks, revealing their strengths, trade-offs, and guiding future improvements in agent-based tools.

Contribution

It offers the first broad comparison of multiple agent frameworks on real-world code-centric tasks using standard benchmarks.

Findings

Agent effectiveness varies with moderate overall success.

AgentOrchestra shows high coordination overhead and longer trajectories.

GPTswarm is the most cost-efficient framework.

Abstract

Unlike traditional automation tools or static LLM-based systems, agents combine decision-making and tool utilization to accomplish complex tasks, showing great potential in software engineering. However, existing studies largely focus on specific tasks or isolated aspects, providing an incomplete picture of agents' practical capabilities. To address this, we conduct a comprehensive empirical study evaluating seven general-purpose agent frameworks across three representative code-centric tasks: software development, vulnerability detection, and program repair. Each task is assessed using standard, widely adopted benchmarks to ensure objective and comparable evaluation. Agent performance is systematically analyzed from three complementary perspectives: effectiveness (task success), efficiency (execution process), and overhead (token consumption). Our findings reveal distinct capability patterns and trade-offs among the evaluated frameworks. In terms of effectiveness, agents achieve moderate overall performance. Regarding efficiency, AgentOrchestra tends to exhibit the longest trajectories and the most correction attempts due to coordination overhead, whereas OpenHands demonstrate stronger reflective reasoning abilities. For overhead, software development incurs the highest monetary cost, while GPTswarm remains the most cost-efficient. Furthermore, we conduct an in-depth cross-analysis of the relationship between effectiveness and efficiency, exploring the underlying reasons behind their interplay. These findings guide both practical adoption and future research toward more efficient software engineering agents.