Turning the Tide: Repository-based Code Reflection

TL;DR

This paper introduces LiveRepoReflection, a challenging benchmark for multi-file repository code understanding, and RepoReflectionCoder, a model trained on a new instruction-tuning dataset, to improve code reflection capabilities in repositories.

Contribution

It presents a novel benchmark and training dataset specifically designed for repository-based code reflection, addressing limitations of previous benchmarks and models.

Findings

Over 40 LLMs evaluated on the new benchmark.

RepoReflectionCoder outperforms existing models in code understanding.

Benchmark reveals challenges in multi-file repository code reflection.

Abstract

Code large language models (LLMs) enhance programming by understanding and generating code across languages, offering intelligent feedback, bug detection, and code updates through reflection, improving development efficiency and accessibility. While benchmarks (e.g. HumanEval/LiveCodeBench) evaluate code generation and real-world relevance, previous works ignore the scenario of modifying code in repositories. Considering challenges remaining in improving reflection capabilities and avoiding data contamination in dynamic benchmarks, we introduce LiveRepoReflection, a challenging benchmark for evaluating code understanding and generation in multi-file repository contexts, featuring 1,888 rigorously filtered test cases across $6$ programming languages to ensure diversity, correctness, and high difficulty. Further, we create RepoReflection-Instruct, a large-scale, quality-filtered instruction-tuning dataset derived from diverse sources, used to train RepoReflectionCoder through a two-turn dialogue process involving code generation and error-driven repair. The leaderboard evaluates over 40 LLMs to reflect the model performance of repository-based code reflection.