Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests

TL;DR

This study compares AI-generated and human-generated pull requests, revealing differences in code change timing, message quality, and communication patterns, and highlighting areas for improving AI coding agents in software development.

Contribution

It provides a comprehensive empirical comparison of agentic versus human pull requests, identifying key differences and suggesting improvements for AI coding agents.

Findings

AI PRs remove symbols faster than human PRs

AI PRs generate stronger commit messages but weaker PR summaries

Commit message length predicts description quality

Abstract

AI coding agents can autonomously generate pull requests (PRs), yet little is known about how their contributions compare to those of humans. We analyze 33,596 agent-generated PRs (APRs) and 6,618 human PRs (HPRs) to compare code-change characteristics and message quality. We observe that APR-introduced symbols (functions and classes) are removed much sooner than those in HPRs (median time to removal 3 vs. 34 days) and are also removed more often (symbol churn 7.33% vs. 4.10%), reflecting a focus on other tasks like documentation and test updates. Agents generate stronger commit-level messages (semantic similarity 0.72 vs. 0.68) but lag humans at PR-level summarization (PR-commit similarity 0.86 vs. 0.88). Commit message length is the best predictor of description quality, indicating reliance on individual commits over full-PR reasoning. These findings highlight a gap between agents' micro-level precision and macro-level communication, suggesting opportunities to improve agent-driven development workflows.