Aligning Offline Metrics and Human Judgments of Value for Code Generation Models

TL;DR

This paper investigates how traditional correctness metrics for code generation models may not fully reflect their practical value, proposing a hybrid metric that better aligns with human judgments of usefulness.

Contribution

The paper introduces a hybrid evaluation metric combining correctness and syntactic similarity, improving correlation with human-perceived value of generated code.

Findings

Correctness alone underestimates the value of code that reduces effort.

Programmers value code that saves effort even if it fails unit tests.

The hybrid metric correlates 14% better with human judgments.

Abstract

Large language models have demonstrated great potential to assist programmers in generating code. For such human-AI pair programming scenarios, we empirically demonstrate that while generated code is most often evaluated in terms of their functional correctness (i.e., whether generations pass available unit tests), correctness does not fully capture (e.g., may underestimate) the productivity gains these models may provide. Through a user study with N = 49 experienced programmers, we show that while correctness captures high-value generations, programmers still rate code that fails unit tests as valuable if it reduces the overall effort needed to complete a coding task. Finally, we propose a hybrid metric that combines functional correctness and syntactic similarity and show that it achieves a 14% stronger correlation with value and can therefore better represent real-world gains when evaluating and comparing models.