Novice Developers Produce Larger Review Overhead for Project Maintainers while Vibe Coding

TL;DR

This study analyzes how AI-assisted vibe coding by novice developers results in larger review overhead and lower acceptance rates compared to experienced developers, impacting project management strategies.

Contribution

It provides empirical evidence on the differences in contribution size and review burden between low- and high-experience vibe coders in AI-assisted development.

Findings

Low-experience vibe coders submit larger PRs with more commits and files.

PRs from low-experience vibe coders receive more review comments and have lower acceptance rates.

Low-experience vibe coders tend to generate more code but shift verification to reviewers.

Abstract

AI coding agents allow software developers to generate code quickly, which raises a practical question for project managers and open source maintainers: can vibe coders with less development experience substitute for expert developers? To explore whether developer experience still matters in AI-assisted development, we study $22,953$ Pull Requests (PRs) from $1,719$ vibe coders in the GitHub repositories of the AIDev dataset. We split vibe coders into lower experience vibe coders ($\mathit{Exp}_{Low}$) and higher experience vibe coders ($\mathit{Exp}_{High}$) and compare contribution magnitude and PR acceptance rates across PR categories. We find that $\mathit{Exp}_{Low}$ submits PRs with larger volume ($2.15\times$ more commits and $1.47\times$ more files changed) than $\mathit{Exp}_{High}$. Moreover, $\mathit{Exp}_{Low}$ PRs, when compared to $\mathit{Exp}_{High}$, receive $4.52\times$ more review comments, and have $31\%$ lower acceptance rates, and remain open $5.16\times$ longer before resolution. Our results indicate that low-experienced vibe coders focus on generating more code while shifting verification burden onto reviewers. For practice, project managers may not be able to safely replace experienced developers with low-experience vibe coders without increasing review capacity. Development teams should therefore combine targeted training for novices with adaptive PR review cycles.