Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension

TL;DR

This paper introduces HuCoSC, a recursive LLM-based method for evaluating code quality by understanding code semantics more accurately, outperforming existing methods in aligning with human judgment and execution results.

Contribution

The paper presents a novel recursive semantic comprehension approach using LLMs and a semantic dependency storage to improve code quality evaluation accuracy.

Findings

HuCoSC outperforms state-of-the-art methods in correlation with human judgment.

HuCoSC achieves higher correlation with code execution results.

The recursive semantic approach enhances LLM understanding of code semantics.

Abstract

Code quality evaluation involves scoring generated code quality based on a reference code for a specific problem statement. Currently, there are two main forms of evaluating code quality: match-based evaluation and execution-based evaluation. The former requires the collection of a large number of test cases, making a huge cost. The latter relies on superficial code matching as an evaluation metric, which fails to accurately capture code semantics. Moreover, extensive research has demonstrated that match-based evaluations do not truly reflect code quality. With the development of large language models (LLMs) in recent years, studies have proven the feasibility of using LLMs as evaluators for generative tasks. However, due to issues like hallucinations and uncertainty in LLMs, their correlation with human judgment remains at a lower level, making the direct use of LLMs for code quality evaluation challenging. To address these issues, we propose Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension (HuCoSC). We employ a recursive approach to enable LLMs to comprehend portions of code semantics independently each time, obtaining the code semantics through multiple interactions with LLMs. We designed a Semantic Dependency Decoupling Storage to make independent analysis feasible, allowing LLMs to achieve more accurate semantics by breaking down complex problems. Finally, the generated code is scored based on a semantic comparison between the reference code and itself. Experimental results indicate that HuCoSC surpasses existing state-of-the-art methods in terms of correlation with human experts and correlation with code execution.