Understanding Dominant Themes in Reviewing Agentic AI-authored Code

TL;DR

This study analyzes how reviewers respond to AI-generated code in open-source projects, identifying key review themes and evaluating LLMs for comment annotation, revealing that reviews focus on documentation, refactoring, styling, testing, and security.

Contribution

It introduces a taxonomy of review comment themes for AI-authored code and demonstrates that LLMs can reliably annotate review comments at scale, aligning well with human judgments.

Findings

Review comments mainly focus on documentation, refactoring, styling, testing, and security.

LLMs achieve high accuracy in annotating review comments, with 78.63% exact match.

AI-generated code reviews still require human oversight for certain aspects.

Abstract

While prior work has examined the generation capabilities of Agentic AI systems, little is known about how reviewers respond to AI-authored code in practice. In this paper, we present a large-scale empirical study of code review dynamics in agent-generated PRs. Using a curated subset of the AIDev dataset, we analyze 19,450 inline review comments spanning 3,177 agent-authored PRs from real-world GitHub repositories. We first derive a taxonomy of 12 review comment themes using topic modeling combined with large language model (LLM)-assisted semantic clustering and consolidation. According to this taxonomy, we then investigate whether zero-shot prompts to LLM can reliably annotate review comments. Our evaluation against human annotations shows that open-source LLM achieves reasonably high exact match (78.63%), macro F1 score (0.78), and substantial agreement with human annotators at the review comment level. At the PR level, the LLM also correctly identifies the dominant review theme with 78% Top-1 accuracy and achieves an average Jaccard similarity of 0.76, indicating strong alignment with human judgments. Applying this annotation pipeline at scale, we find that apart from functional correctness and logical changes, reviews of agent-authored PRs predominantly focus on documentation gaps, refactoring needs, styling and formatting issues, with testing and security-related concerns. These findings suggest that while AI agents can accelerate code production, there remain gaps requiring targeted human review oversight.