Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects

TL;DR

This paper introduces a benchmark and an agent for automating the installation of Python repositories using LLMs, highlighting current capabilities and challenges in autonomous software engineering tasks.

Contribution

It presents a new benchmark for repository installation tasks and the Installamatic agent, advancing the evaluation of LLMs in automating dependency management.

Findings

55% of repositories can be installed at least once by the agent

Identified key challenges in automating repository installation

Provided insights into improving LLM-based software engineering agents

Abstract

Many works have recently proposed the use of Large Language Model (LLM) based agents for performing `repository level' tasks, loosely defined as a set of tasks whose scopes are greater than a single file. This has led to speculation that the orchestration of these repository-level tasks could lead to software engineering agents capable of performing almost independently of human intervention. However, of the suite of tasks that would need to be performed by this autonomous software engineering agent, we argue that one important task is missing, which is to fulfil project level dependency by installing other repositories. To investigate the feasibility of this repository level installation task, we introduce a benchmark of of repository installation tasks curated from 40 open source Python projects, which includes a ground truth installation process for each target repository. Further, we propose Installamatic, an agent which aims to perform and verify the installation of a given repository by searching for relevant instructions from documentation in the repository. Empirical experiments reveal that that 55% of the studied repositories can be automatically installed by our agent at least one out of ten times. Through further analysis, we identify the common causes for our agent's inability to install a repository, discuss the challenges faced in the design and implementation of such an agent and consider the implications that such an agent could have for developers.