Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration

TL;DR

Alibaba LingmaAgent is a novel automated issue resolution method that leverages comprehensive repository exploration and knowledge graphs, significantly improving accuracy and efficiency in real-world software engineering tasks.

Contribution

The paper introduces LingmaAgent, a new approach that uses repository-level knowledge graphs and Monte Carlo tree search to enhance automated issue resolution beyond local code analysis.

Findings

Achieved 18.5% improvement on SWE-bench Lite benchmark.

Automatically resolved 16.9% of in-house issues at Alibaba Cloud.

Solved 43.3% of problems after manual intervention.

Abstract

This paper presents Alibaba LingmaAgent, a novel Automated Software Engineering method designed to comprehensively understand and utilize whole software repositories for issue resolution. Deployed in TONGYI Lingma, an IDE-based coding assistant developed by Alibaba Cloud, LingmaAgent addresses the limitations of existing LLM-based agents that primarily focus on local code information. Our approach introduces a top-down method to condense critical repository information into a knowledge graph, reducing complexity, and employs a Monte Carlo tree search based strategy enabling agents to explore and understand entire repositories. We guide agents to summarize, analyze, and plan using repository-level knowledge, allowing them to dynamically acquire information and generate patches for real-world GitHub issues. In extensive experiments, LingmaAgent demonstrated significant improvements, achieving an 18.5\% relative improvement on the SWE-bench Lite benchmark compared to SWE-agent. In production deployment and evaluation at Alibaba Cloud, LingmaAgent automatically resolved 16.9\% of in-house issues faced by development engineers, and solved 43.3\% of problems after manual intervention. Additionally, we have open-sourced a Python prototype of LingmaAgent for reference by other industrial developers https://github.com/RepoUnderstander/RepoUnderstander. In fact, LingmaAgent has been used as a developed reference by many subsequently agents.