Evaluating LLM-Generated Code: A Benchmark and Developer Study

TL;DR

This paper introduces a comprehensive evaluation methodology for LLM-generated code that assesses correctness, quality, and developer opinions, demonstrated through comparing three major LLMs.

Contribution

It presents a novel multi-level evaluation framework combining correctness, quality, and developer feedback for code generated by large language models.

Findings

Developer reviews reveal insights into production readiness of code.

The methodology uncovers aspects not captured by standard correctness benchmarks.

Comparative analysis of GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4 shows differing strengths.

Abstract

Code generation is one of the tasks for which the use of Large Language Models is widely adopted and highly successful. Given this popularity, there are many benchmarks dedicated to code generation that can help select the best model. However, they primarily focus on measuring solution correctness, leaving other aspects, such as code quality and usability, behind. This paper aims to describe a custom tree-fold evaluation methodology for code generated by Large Language Models that bridges this gap. The methodology includes a dedicated correctness benchmark based on a complex multi-level computer science project, code quality verification, and a survey of developers' opinions on generated code samples gathered through a structured code-review process. The proposed methodology's usage and usefulness are demonstrated by evaluating and comparing three general-purpose Large Language Models: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. The results show that reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach.