On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories

TL;DR

This paper empirically examines how reviewer bot feedback influences pull request outcomes in GitHub, highlighting that high-relevance feedback improves workflow efficiency more than comment volume.

Contribution

It provides the first large-scale analysis of reviewer bot comments, revealing the impact of feedback quality and activity volume on PR acceptance and resolution times.

Findings

Reviewer bots mainly focus on bug fixes, testing, and documentation.

Higher comment volume correlates with longer PR resolution times.

Feedback relevance has no significant impact on workflow outcomes.

Abstract

Autonomous coding agents are reshaping software development by creating pull requests (PRs) on GitHub, referred to as agentic PRs. In parallel, the review process is also becoming autonomous, thereby making reviewer bots key actors in the assessment of these agentic PRs. However, their influence on PR acceptance and resolution remains unclear. This study empirically investigates the relationship between reviewer-bot feedback and PR outcomes by analyzing how Reviewer Bot Feedback Quality (relevance, clarity, conciseness) and Reviewer Bot Activity Volume (comment count) are associated with PR acceptance and resolution time. We analyze 7,416 reviewer-bot comments on 4,532 PRs from the AI_Dev dataset (a dataset that captured AI agents' PRs in GitHub projects). Our results show that reviewer-bot comments mainly focus on bug fixes, testing, and documentation, are civil in tone, and are prescriptive in nature. Reviewer bots generally produce clear and concise feedback, though the semantic relevance of comments to underlying code changes is moderate. We find that higher Reviewer Bot Activity volume is associated with longer PR resolution times and lower average feedback quality, showing that as bots generate more comments on a PR, the average pertinence of that feedback appears to degrade. At the same time, Reviewer Bot Feedback Quality shows no meaningful association with workflow outcomes. Our findings suggest that, in agentic PR workflows, reviewer bots should prioritize targeted high-relevance feedback over generating large numbers of comments.