RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices

TL;DR

RealBench introduces a benchmark for repo-level code generation that uses structured system designs like UML diagrams, better reflecting real-world software development practices and evaluating LLMs' capabilities in this context.

Contribution

This paper presents a new benchmark aligned with industry practices, incorporating structured designs, and systematically evaluates LLMs' performance in repo-level code generation.

Findings

LLMs perform worse at repo-level code generation compared to module-level tasks.

LLMs excel at creating modules from UML diagrams but often produce poor quality code.

Generating entire repositories at once is effective for small projects, while module-by-module works better for complex repositories.

Abstract

Writing code requires significant time and effort in software development. To automate this process, researchers have made substantial progress using Large Language Models (LLMs) for code generation. Many benchmarks like HumanEval and EvoCodeBench have been created to evaluate LLMs by requiring them to generate code from natural language requirements. However, in enterprise applications and team development, developers typically write code based on structured designs or specifications rather than raw natural language descriptions. This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate software development tasks. To address this gap, we propose RealBench, a repository-level code generation benchmark aligned with real-world industry software development practices. Each example includes both natural language requirements and UML diagrams as system design, matching how developers typically receive specifications. Based on the constructed benchmarks, we conduct a systematic evaluation of advanced LLMs' code generation capabilities when provided with structured system designs. The experimental results reveal key insights in current LLMs' capabilities for repo-level code generation aligned with real-world software development practices. First, we notice that regarding repo-level code generation, LLMs show much worse performance and there are significant performance gaps among LLMs. Second, LLMs are good at finding and creating modules defined in UML diagrams, but the quality of generated modules is often poor due to grammar and logic errors. Third, generating the entire repository at once is the best generation strategy on smaller repositories, while generating a complex repository with the module-by-module strategy works better compared to other strategies.