Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents

TL;DR

This study compares rejection reasons across different autonomous coding agents and humans, revealing agent-specific rejection patterns and proposing heuristics to better understand and analyze PR rejections in agentic coding workflows.

Contribution

It provides a comparative analysis of rejection reasons for multiple coding agents, highlighting agent-specific failure modes and introducing heuristics to improve rejection reason identification.

Findings

Seven rejection modes are unique to agentic PRs, including distrust of AI-generated code.

Agent-specific rejection patterns reflect differences in agent configuration and usage.

A large portion of rejected PRs lack explicit reviewer feedback, complicating rejection analysis.

Abstract

Agentic coding -- software development workflows in which autonomous coding agents plan, implement, and submit code changes with minimal human involvement -- is rapidly gaining traction. Prior work has shown that Pull Requests (PRs) produced using coding agents (Agentic-PRs) are accepted less often than PRs that are not labeled as agentic (Human-PRs). The rejection reasons for a single agent (Claude Code) have been explored, but a comparison of how rejection reasons differ between Agentic-PRs generated by different agents has not yet been performed. This comparison is important since different coding agents are often used for different purposes, which can lead to agent-specific failure patterns. In this paper, we inspect 654 rejected PRs from the AIDev dataset covering five coding agents, as well as a human baseline. Our results show that seven rejection modes occur only in Agentic-PRs, including distrust of AI-generated code. We also observe agent-specific patterns (e.g., automated withdrawal of inactive PRs by Devin), reflecting differences in how agents are configured and used in practice. Notably, a large proportion of rejected PRs (67.9%) lack explicit reviewer feedback, making their rejection reasons difficult to determine. To mitigate this issue, we propose a set of heuristics that reduce the proportion of such cases, offering a practical preprocessing step for future studies of PR rejection in agentic coding.