What's Wrong with Your Code Generated by Large Language Models? An Extensive Study

TL;DR

This extensive empirical study evaluates the performance and limitations of various large language models in code generation, identifying challenges with complex problems, bug types, and proposing a self-critique correction method.

Contribution

The paper provides a comprehensive analysis of LLMs' code generation capabilities, introduces a bug taxonomy, and proposes a novel self-critique method to improve code quality without additional training.

Findings

LLMs struggle with complex problems, producing shorter yet more complicated code.

A bug taxonomy with 3 categories and 10 sub-categories was developed and analyzed.

A self-critique iterative method improves code correctness without retraining.

Abstract

The increasing development of LLMs in code generation has drawn significant attention among researchers. To enhance LLM-based code generation ability, current efforts are predominantly directed towards collecting high-quality datasets and leveraging diverse training technologies. However, there is a notable lack of comprehensive studies examining the limitations and boundaries of existing methods. To bridge this gap, we conducted an extensive empirical study evaluating the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks. Our investigation, which evaluated the length, cyclomatic complexity and API number of the generated code, revealed that these LLMs face challenges in generating successful code for more complex problems, and tend to produce code that is shorter yet more complicated as compared to canonical solutions. Additionally, we developed a taxonomy of bugs for incorrect codes that includes three categories and ten sub-categories, and analyzed the root cause for common bug types. To better understand the performance of LLMs in real-world projects, we also manually created a real-world benchmark RWPB. We analyzed bugs on RWPB to highlight distinct differences in bug distributions between actual scenarios and existing benchmarks. Finally, we propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback. Our comprehensive and extensive study provides insights into the current limitations of LLM-based code generation and opportunities for enhancing the accuracy and quality of the generated code.