Large Language Model Critics for Execution-Free Evaluation of Code Changes

TL;DR

This paper introduces LLM-based critics that provide execution-free, step-level evaluation of code changes using a reference patch, significantly improving assessment accuracy over existing metrics and aiding in comparing different code generation workflows.

Contribution

The paper presents a novel reference-aware LLM critic framework for evaluating code changes without execution, outperforming existing critics and enabling better comparison of agentic workflows.

Findings

Achieved 91.6% F1 score in predicting executability.

Predicted build status with 84.8% accuracy.

Outperformed other critics by 38.9% to 72.5%.

Abstract

Large language models (LLMs) offer a promising way forward for automating software engineering tasks, such as bug fixes, feature additions, etc., via multi-step LLM-based agentic workflows. However, existing metrics for evaluating such workflows, mainly build status and occasionally log analysis, are too sparse and limited in providing the information needed to assess the quality of changes made. In this work, we designed LLM-based critics to derive well-structured and rigorous intermediate/step-level, execution-free evaluation proxies for repo-level code changes. Importantly, we assume access to the gold test patch for the problem (i.e., reference-aware) to assess both semantics and executability of generated patches. With the gold test patch as a reference, we predict executability of all editing locations with an F1 score of 91.6%, aggregating which, we can predict the build status in 84.8% of the instances in SWE-bench. In particular, such an execution-focused LLM critic outperforms other reference-free and reference-aware LLM critics by 38.9% to 72.5%. Moreover, we demonstrate the usefulness of such a reference-aware framework in comparing patches generated by different agentic workflows. Finally, we open-source the library developed for this project, which allows further usage for either other agentic workflows or other benchmarks. The source code is available at https://github.com/amazon-science/code-agent-eval.