Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'

TL;DR

This paper introduces REPOCOD, a challenging real-world Python code-generation benchmark with complex dependencies, revealing current LLMs' limited performance and highlighting the need for more advanced models to assist developers effectively.

Contribution

The paper presents REPOCOD, a new benchmark with realistic, large-scale project tasks and evaluation metrics, addressing limitations of previous benchmarks in assessing LLMs' real-world coding abilities.

Findings

No LLM exceeds 30% pass@1 on REPOCOD

Retrieval-augmented generation outperforms context-based methods

Current LLMs are insufficient for real-world software development

Abstract

Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.