Is LLM-Generated Code More Maintainable \& Reliable than Human-Written Code?

TL;DR

This study empirically compares the maintainability and reliability of LLM-generated code to human-written code across various difficulty levels, revealing that LLMs often produce less bug-prone code but can introduce structural issues in complex tasks.

Contribution

It provides a comprehensive empirical analysis of LLM-generated code quality, highlighting strengths in bug reduction and limitations in structural correctness for complex problems.

Findings

LLM-generated code has fewer bugs and requires less effort to fix.

Fine-tuning reduces high-severity issues but may decrease overall performance.

In complex tasks, LLM solutions can introduce structural issues not seen in human code.

Abstract

Background: The rise of Large Language Models (LLMs) in software development has opened new possibilities for code generation. Despite the widespread use of this technology, it remains unclear how well LLMs generate code solutions in terms of software quality and how they compare to human-written code. Aims: This study compares the internal quality attributes of LLM-generated and human-written code. Method: Our empirical study integrates datasets of coding tasks, three LLM configurations (zero-shot, few-shot, and fine-tuning), and SonarQube to assess software quality. The dataset comprises Python code solutions across three difficulty levels: introductory, interview, and competition. We analyzed key code quality metrics, including maintainability and reliability, and the estimated effort required to resolve code issues. Results: Our analysis shows that LLM-generated code has fewer bugs and requires less effort to fix them overall. Interestingly, fine-tuned models reduced the prevalence of high-severity issues, such as blocker and critical bugs, and shifted them to lower-severity categories, but decreased the model's performance. In competition-level problems, the LLM solutions sometimes introduce structural issues that are not present in human-written code. Conclusion: Our findings provide valuable insights into the quality of LLM-generated code; however, the introduction of critical issues in more complex scenarios highlights the need for a systematic evaluation and validation of LLM solutions. Our work deepens the understanding of the strengths and limitations of LLMs for code generation.