A Survey of Bugs in AI-Generated Code

TL;DR

This paper systematically reviews the types, distribution, and mitigation strategies of bugs in AI-generated code, providing a comprehensive understanding to guide future improvements and quality assessment.

Contribution

It offers the first systematic classification and analysis of bugs in AI-generated code, summarizing existing literature and identifying key error patterns and remediation strategies.

Findings

Bugs in AI-generated code are diverse and include common defect types.

Certain models are more prone to specific bug patterns.

Effective mitigation strategies are identified and discussed.

Abstract

Developers are widely using AI code-generation models, aiming to increase productivity and efficiency. However, there are also quality concerns regarding the AI-generated code. The generated code is produced by models trained on publicly available code, which are known to contain bugs and quality issues. Those issues can cause trust and maintenance challenges during the development process. Several quality issues associated with AI-generated code have been reported, including bugs and defects. However, these findings are often scattered and lack a systematic summary. A comprehensive review is currently lacking to reveal the types and distribution of these errors, possible remediation strategies, as well as their correlation with the specific models. In this paper, we systematically analyze the existing AI-generated code literature to establish an overall understanding of bugs and defects in generated code, providing a reference for future model improvement and quality assessment. We aim to understand the nature and extent of bugs in AI-generated code, and provide a classification of bug types and patterns present in code generated by different models. We also discuss possible fixes and mitigation strategies adopted to eliminate bugs from the generated code.