Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity

TL;DR

This large-scale study compares human-written and AI-generated code across defect rates, vulnerabilities, and complexity, revealing distinct quality profiles and highlighting the need for tailored assurance practices in AI-assisted software development.

Contribution

It provides a comprehensive analysis of AI versus human code quality across multiple dimensions and languages, using a large dataset and standardized classification methods.

Findings

AI code is simpler but more prone to vulnerabilities.

Human code has greater complexity and maintainability issues.

AI code contains more high-risk security vulnerabilities.

Abstract

As AI code assistants become increasingly integrated into software development workflows, understanding how their code compares to human-written programs is critical for ensuring reliability, maintainability, and security. In this paper, we present a large-scale comparison of code authored by human developers and three state-of-the-art LLMs, i.e., ChatGPT, DeepSeek-Coder, and Qwen-Coder, on multiple dimensions of software quality: code defects, security vulnerabilities, and structural complexity. Our evaluation spans over 500k code samples in two widely used languages, Python and Java, classifying defects via Orthogonal Defect Classification and security vulnerabilities using the Common Weakness Enumeration. We find that AI-generated code is generally simpler and more repetitive, yet more prone to unused constructs and hardcoded debugging, while human-written code exhibits greater structural complexity and a higher concentration of maintainability issues. Notably, AI-generated code also contains more high-risk security vulnerabilities. These findings highlight the distinct defect profiles of AI- and human-authored code and underscore the need for specialized quality assurance practices in AI-assisted programming.