How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests

TL;DR

This large-scale study compares AI-generated and human-generated pull requests on GitHub, revealing significant differences in commit patterns and slight differences in description accuracy, informing their impact on software development.

Contribution

It provides the first comprehensive empirical analysis of AI coding agents' contributions to open source, highlighting key differences from human developers.

Findings

AI PRs differ significantly in commit count from human PRs.

AI PRs show moderate differences in files touched and lines deleted.

AI PR descriptions have slightly higher similarity to diffs than human PRs.

Abstract

AI coding agents are increasingly acting as autonomous contributors by generating and submitting pull requests (PRs). However, we lack empirical evidence on how these agent-generated PRs differ from human contributions, particularly in how they modify code and describe their changes. Understanding these differences is essential for assessing their reliability and impact on development workflows. Using the MSR 2026 Mining Challenge version of the AIDev dataset, we analyze 24,014 merged Agentic PRs (440,295 commits) and 5,081 merged Human PRs (23,242 commits). We examine additions, deletions, commits, and files touched, and evaluate the consistency between PR descriptions and their diffs using lexical and semantic similarity. Agentic PRs differ substantially from Human PRs in commit count (Cliff's $\delta = 0.5429$) and show moderate differences in files touched and deleted lines. They also exhibit slightly higher description-to-diff similarity across all measures. These findings provide a large-scale empirical characterization of how AI coding agents contribute to open source development.