The Fault in our Stars: Quality Assessment of Code Generation Benchmarks

TL;DR

This study critically evaluates the quality of prompts in code generation benchmarks, revealing prevalent issues and demonstrating that prompt improvements can enhance Python code generation performance, while also uncovering potential data contamination in models.

Contribution

It is the first comprehensive analysis of prompt quality in code generation benchmarks, highlighting common issues and their impact on model evaluation.

Findings

Benchmark prompts often contain spelling and grammatical errors.

Fixing prompt issues improves Python code generation performance.

Evidence of data contamination in GPT-3.5-Turbo and CodeGen-2.5 models.

Abstract

Large Language Models (LLMs) are gaining popularity among software engineers. A crucial aspect of developing effective code generation LLMs is to evaluate these models using a robust benchmark. Evaluation benchmarks with quality issues can provide a false sense of performance. In this work, we conduct the first-of-its-kind study of the quality of prompts within benchmarks used to compare the performance of different code generation models. To conduct this study, we analyzed 3,566 prompts from 9 code generation benchmarks to identify quality issues in them. We also investigated whether fixing the identified quality issues in the benchmarks' prompts affects a model's performance. We also studied memorization issues of the evaluation dataset, which can put into question a benchmark's trustworthiness. We found that code generation evaluation benchmarks mainly focused on Python and coding exercises and had very limited contextual dependencies to challenge the model. These datasets and the developers' prompts suffer from quality issues like spelling and grammatical errors, unclear sentences to express developers' intent, and not using proper documentation style. Fixing all these issues in the benchmarks can lead to a better performance for Python code generation, but not a significant improvement was observed for Java code generation. We also found evidence that GPT-3.5-Turbo and CodeGen-2.5 models may have data contamination issues.