ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation

TL;DR

This paper introduces a time-consistent benchmark for evaluating repository-aware software engineering systems, emphasizing temporal validity and prompt construction effects on model performance.

Contribution

It proposes a novel benchmark methodology that ensures temporal consistency and controls prompt variables, enabling more valid evaluation of software engineering models.

Findings

File-level F1 reaches over 0.80 with guided prompts.

Prompt construction significantly impacts model performance.

Temporal consistency is crucial for valid repository-aware evaluation.

Abstract

Evaluation of repository-aware software engineering systems is often confounded by synthetic task design, prompt leakage, and temporal contamination between repository knowledge and future code changes. We present a time-consistent benchmark methodology that snapshots a repository at time T0, constructs repository-derived code knowledge using only artifacts available before T0, and evaluates on engineering tasks derived from pull requests merged in the future interval (T0, T1]. Each historical pull request is transformed into a natural-language task through an LLM-assisted prompt-generation pipeline, and the benchmark is formalized as a matched A/B comparison in which the same software engineering agent is evaluated with and without repository-derived code knowledge while all other variables are held constant. We also report a baseline characterization study on two open-source repositories, DragonFly and React, using three Claude-family models and four prompt granularities. Across both repositories, file-level F1 increases monotonically from minimal to guided prompts, reaching 0.8081 on DragonFly and 0.8078 on React for the strongest tested model. These results show that prompt construction is a first-order benchmark variable. More broadly, the benchmark highlights that temporal consistency and prompt control are core validity requirements for repository-aware software engineering evaluation.