CodeScore-R: An Automated Robustness Metric for Assessing the FunctionalCorrectness of Code Synthesis

TL;DR

CodeScore-R is an automated, robust evaluation metric for code synthesis that accurately assesses functional correctness without test cases, using contrastive learning and transformations to ensure stability against minor code changes.

Contribution

The paper introduces CodeScore-R, a novel automated metric combining contrastive learning and transformations to evaluate code functionality more robustly and efficiently.

Findings

Outperforms existing metrics in code generation and migration tasks

More closely aligned with Pass@k metric

Demonstrates stronger robustness against code variations

Abstract

Evaluation metrics are crucial in the field of code synthesis. Commonly used code evaluation metrics canbe classified into three types: match-based, semantic-based, and execution-based. Among them, the execution-basedPass@k metric accurately assesses the functionality of predicted code by executing test cases. However, calculatingthis metric requires a significant amount of overhead, necessitating the design of an automated evaluation metric thatcan assess the functionality of predicted code without the need for test cases. Additionally, a good evaluation metricshould be robust, that is the metric can maintain its accuracy even when the predicted code undergoes minor changes.To address these challenges, we propose an automated robust metric, called CodeScore-R, based on UniXcoder andcontrastive learning, for evaluating the functionality of code synthesis. CodeScore-R employs techniques such assketch-based processing, syntactic-equivalent transformations, and mutation testing to effectively mitigate theinterference caused by identifiers, syntax structures, and operators on evaluation results. Experimental resultsdemonstrate that in the tasks of code generation and migration in Java and Python, CodeScore-R outperforms otherevaluation metrics and is more closely aligned with the Pass@k metric, while exhibiting stronger robustness.