Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study

TL;DR

This empirical study investigates why AI-generated fix-related pull requests often remain unmerged, analyzing 8,106 PRs to identify common failure reasons and highlight limitations of current AI coding agents in real-world software development.

Contribution

The paper provides the first comprehensive empirical analysis of AI agent involved fix PRs, identifying key failure reasons and offering insights for improving AI-human collaboration in software maintenance.

Findings

Test case failures are the most common reason for PR rejection.

Prior resolution of issues by other PRs often prevents merging.

Build or deployment failures are relatively rare.

Abstract

Autonomous coding agents (e.g., OpenAI Codex, Devin, GitHub Copilot) are increasingly used to generate fix-related pull requests (PRs) in real world software repositories. However, their practical effectiveness depends on whether these contributions are accepted and merged by project maintainers. In this paper, we present an empirical study of AI agent involved fix related PRs, examining both their integration outcomes, latency, and the factors that hinder successful merging. We first analyze 8,106 fix related PRs authored by five widely used AI coding agents from the AIDEV POP dataset to quantify the proportions of PRs that are merged, closed without merging, or remain open. We then conduct a manual qualitative analysis of a statistically significant sample of 326 closed but unmerged PRs, spending approximately 100 person hours to construct a structured catalog of 12 failure reasons. Our results indicate that test case failures and prior resolution of the same issues by other PRs are the most common causes of non integration, whereas build or deployment failures are comparatively rare. Overall, our findings expose key limitations of current AI coding agents in real world settings and highlight directions for their further improvement and for more effective human AI collaboration in software maintenance.