Let's Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests

TL;DR

This paper empirically analyzes 40,214 pull requests to understand how human and AI-generated PRs differ in merge success, revealing key features influencing outcomes and guiding improvements in AI-assisted development.

Contribution

It provides a large-scale empirical comparison of human and AI-generated pull requests, identifying factors affecting merge rates and suggesting ways to enhance PR quality through collaboration.

Findings

Submitter attributes significantly influence merge outcomes.

Review features have contrasting effects on human vs. AI PRs.

Insights support improving AI-assisted pull request quality.

Abstract

The automatic generation of pull requests (PRs) using AI agents has become increasingly common. Although AI-generated PRs are fast and easy to create, their merge rates have been reported to be lower than those created by humans. In this study, we conduct a large-scale empirical analysis of 40,214 PRs collected from the AIDev dataset. We extract 64 features across six families and fit statistical regression models to compare PR merge outcomes for human and agentic PRs, as well as across three AI agents. Our results show that submitter attributes dominate merge outcomes for both groups, while review-related features exhibit contrasting effects between human and agentic PRs. The findings of this study provide insights into improving PR quality through human-AI collaboration.