Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
Khairul Alam, Saikat Mondal, Banani Roy

TL;DR
This empirical study investigates why AI-generated fix-related pull requests often remain unmerged, analyzing 8,106 PRs to identify common failure reasons and highlight limitations of current AI coding agents in real-world software development.
Contribution
The paper provides the first comprehensive empirical analysis of AI agent involved fix PRs, identifying key failure reasons and offering insights for improving AI-human collaboration in software maintenance.
Findings
Test case failures are the most common reason for PR rejection.
Prior resolution of issues by other PRs often prevents merging.
Build or deployment failures are relatively rare.
Abstract
Autonomous coding agents (e.g., OpenAI Codex, Devin, GitHub Copilot) are increasingly used to generate fix-related pull requests (PRs) in real world software repositories. However, their practical effectiveness depends on whether these contributions are accepted and merged by project maintainers. In this paper, we present an empirical study of AI agent involved fix related PRs, examining both their integration outcomes, latency, and the factors that hinder successful merging. We first analyze 8,106 fix related PRs authored by five widely used AI coding agents from the AIDEV POP dataset to quantify the proportions of PRs that are merged, closed without merging, or remain open. We then conduct a manual qualitative analysis of a statistically significant sample of 326 closed but unmerged PRs, spending approximately 100 person hours to construct a structured catalog of 12 failure reasons.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Scientific Computing and Data Management · Software Engineering Techniques and Practices
