Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs

TL;DR

This study compares the frequency and context of breaking changes in AI-generated versus human-authored pull requests in Python open-source projects, revealing lower overall risk but higher risk during maintenance tasks.

Contribution

It provides a comprehensive analysis of breaking change patterns in AI versus human PRs, introducing a tool for AST-based detection and highlighting the 'Confidence Trap' phenomenon.

Findings

AI agents introduce fewer breaking changes overall than humans (3.45% vs. 7.40%).

Higher risk of breaking changes during maintenance tasks like refactoring and chores.

Highly confident AI PRs can still cause breaking changes, indicating need for stricter review.

Abstract

AI coding agents are increasingly integrated into modern software engineering workflows, actively collaborating with human developers to create pull requests (PRs) in open-source repositories. Although coding agents improve developer productivity, they often generate code with more bugs and security issues than human-authored code. While human-authored PRs often break backward compatibility, leading to breaking changes, the potential for agentic PRs to introduce breaking changes remains underexplored. The goal of this paper is to help developers and researchers evaluate the reliability of AI-generated PRs by examining the frequency and task contexts in which AI agents introduce breaking changes. We conduct a comparative analysis of 7,191 agent-generated PRs with 1402 human-authored PRs from Python repositories in the AIDev dataset. We develop a tool that analyzes code changes in commits corresponding to the agentic PRs and leverages an abstract syntax tree (AST) based analysis to detect potential breaking changes. Our findings show that AI agents introduce fewer breaking changes overall than humans (3.45% vs. 7.40%) in code generation tasks. However, agents exhibit substantially higher risk during maintenance tasks, with refactoring and chore changes introducing breaking changes at rates of 6.72% and 9.35%, respectively. We also identify a "Confidence Trap" where highly confident agentic PRs still introduce breaking changes, indicating the need for stricter review during maintenance oriented changes regardless of reported confidence score.