On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generation

TL;DR

Embedding-based metrics like CodeBERTScore are weakly correlated with true functional correctness in code generation but strongly correlate with editing effort, highlighting their limitations for evaluating code quality.

Contribution

This work critically evaluates the effectiveness of embedding-based metrics for measuring functional correctness in code generation, revealing their limitations.

Findings

Weak correlation (0.16) with functional correctness.

Strong correlation (0.72) with editing effort.

Highlights need for better evaluation metrics.

Abstract

The task of code generation from natural language (NL2Code) has become extremely popular, especially with the advent of Large Language Models (LLMs). However, efforts to quantify and track this progress have suffered due to a lack of reliable metrics for functional correctness. While popular benchmarks like HumanEval have test cases to enable reliable evaluation of correctness, it is time-consuming and requires human effort to collect test cases. As an alternative several reference-based evaluation metrics have been proposed, with embedding-based metrics like CodeBERTScore being touted as having a high correlation with human preferences and functional correctness. In our work, we analyze the ability of embedding-based metrics like CodeBERTScore to measure functional correctness and other helpful constructs like editing effort by analyzing outputs of ten models over two popular code generation benchmarks. Our results show that while they have a weak correlation with functional correctness (0.16), they are strongly correlated (0.72) with editing effort.