Repo2Run: Automated Building Executable Environment for Code Repository at Scale

TL;DR

Repo2Run is an innovative LLM-based system that automates the creation of executable test environments for code repositories, significantly enhancing scalability and success rates in building Docker environments for diverse repositories.

Contribution

It introduces the first LLM-driven approach to automate building Docker environments for repositories at scale, reducing manual effort and increasing success rates.

Findings

Achieves 86.0% success rate in building environments

Outperforms previous methods by 77.0%

Successfully automates Dockerfile synthesis for Python repositories

Abstract

Scaling up executable code data is significant for improving language models' software engineering capability. The intricate nature of the process makes it labor-intensive, time-consuming and expert-knowledge-dependent to build a large number of executable code repositories, limiting the scalability of existing work based on running tests. The primary bottleneck lies in the automated building of test environments for different repositories, which is an essential yet underexplored task. To mitigate the gap, we introduce Repo2Run, the first LLM-based agent aiming at automating the building of executable test environments for any repositories at scale. Specifically, given a code repository, Repo2Run iteratively builds the Docker image, runs unit tests based on the feedback of the building, and synthesizes the Dockerfile until the entire pipeline is executed successfully. The resulting Dockerfile can then be used to create Docker container environments for running code and tests. We created a benchmark containing 420 Python repositories with unit tests for evaluation. The results illustrate that Repo2Run achieves an 86.0% success rate, outperforming SWE-agent by 77.0%. The resources of Repo2Run are available at https://github.com/bytedance/Repo2Run.