CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation

TL;DR

This paper introduces CODE-DITING, a new reasoning-based metric for evaluating code that balances accuracy, efficiency, and explainability, outperforming larger models and existing methods.

Contribution

The paper proposes CODE-DITING, a novel, scalable, and explainable code evaluation method that transfers reasoning capabilities from large models to smaller ones, improving performance and reducing costs.

Findings

CODE-DITING 1.5B outperforms similarly sized models.

CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B.

The method is robust to preference leakage.

Abstract

Trustworthy evaluation methods for code snippets play a crucial role in neural code generation. Traditional methods, which either rely on reference solutions or require executable test cases, have inherent limitation in flexibility and scalability. The recent LLM-as-Judge methodology offers a promising alternative by directly evaluating functional consistency between the problem description and the generated code. To systematically understand the landscape of these LLM-as-Judge methods, we conduct a comprehensive empirical study across three diverse datasets. Our investigation reveals the pros and cons of two categories of LLM-as-Judge methods: the methods based on general foundation models can achieve good performance but require complex prompts and lack explainability, while the methods based on reasoning foundation models provide better explainability with simpler prompts but demand substantial computational resources due to their large parameter sizes. To address these limitations, we propose CODE-DITING, a novel code evaluation method that balances accuracy, efficiency and explainability. We develop a data distillation framework that effectively transfers reasoning capabilities from DeepSeek-R1671B to our CODE-DITING 1.5B and 7B models, significantly enhancing evaluation explainability and reducing the computational cost. With the majority vote strategy in the inference process, CODE-DITING 1.5B outperforms all models with the same magnitude of parameters and achieves performance which would normally exhibit in a model with 5 times of parameter scale. CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B, even though it only uses 1% of the parameter volume of these large models. Further experiments show that CODEDITING is robust to preference leakage and can serve as a promising alternative for code evaluation.