Comment Traps: How Defective Commented-out Code Augment Defects in AI-Assisted Code Generation

TL;DR

This paper investigates how defective commented-out code influences AI code generation tools, revealing that such code significantly increases defects in generated code and highlighting the need for more robust AI assistant safeguards.

Contribution

It is the first study to systematically evaluate the impact of defective commented-out code on AI code assistants like GitHub Copilot and Cursor.

Findings

Defective commented-out code causes up to 58.17% more defects in generated code.

AI tools actively reason through defective patterns rather than copying them.

Explicit instructions to ignore defective code reduce defects by only 21.84%.

Abstract

With the rapid development of large language models in code generation, AI-powered editors such as GitHub Copilot and Cursor are revolutionizing software development practices. At the same time, studies have identified potential defects in the generated code. Previous research has predominantly examined how code context influences the generation of defective code, often overlooking the impact of defects within commented-out code (CO code). AI coding assistants' interpretation of CO code in prompts affects the code they generate. This study evaluates how AI coding assistants, GitHub Copilot and Cursor, are influenced by defective CO code. The experimental results show that defective CO code in the context causes AI coding assistants to generate more defective code, reaching up to 58.17 percent. Our findings further demonstrate that the tools do not simply copy the defective code from the context. Instead, they actively reason to complete incomplete defect patterns and continue to produce defective code despite distractions such as incorrect indentation or tags. Even with explicit instructions to ignore the defective CO code, the reduction in defects does not exceed 21.84 percent. These findings underscore the need for improved robustness and security measures in AI coding assistants.