CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories

TL;DR

This paper introduces CSR-Bench, a benchmark for evaluating LLMs in deploying computer science research repositories, and presents CSR-Agents, a framework for automating repository deployment to improve research workflow efficiency.

Contribution

The paper presents CSR-Bench for assessing LLMs in research deployment and introduces CSR-Agents, a novel multi-agent framework for automating code repository deployment.

Findings

LLM agents can automate repository deployment tasks effectively.

Preliminary results show increased productivity in research workflows.

CSR-Bench provides a comprehensive evaluation of LLM capabilities in research settings.

Abstract

The increasing complexity of computer science research projects demands more effective tools for deploying code repositories. Large Language Models (LLMs), such as Anthropic Claude and Meta Llama, have demonstrated significant advancements across various fields of computer science research, including the automation of diverse software engineering tasks. To evaluate the effectiveness of LLMs in handling complex code development tasks of research projects, particularly for NLP/CV/AI/ML/DM topics, we introduce CSR-Bench, a benchmark for Computer Science Research projects. This benchmark assesses LLMs from various aspects including accuracy, efficiency, and deployment script quality, aiming to explore their potential in conducting computer science research autonomously. We also introduce a novel framework, CSR-Agents, that utilizes multiple LLM agents to automate the deployment of GitHub code repositories of computer science research projects. Specifically, by checking instructions from markdown files and interpreting repository structures, the model generates and iteratively improves bash commands that set up the experimental environments and deploy the code to conduct research tasks. Preliminary results from CSR-Bench indicate that LLM agents can significantly enhance the workflow of repository deployment, thereby boosting developer productivity and improving the management of developmental workflows.