Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests

TL;DR

This study investigates the inconsistency between AI-generated pull request descriptions and actual code changes, revealing significant trust issues and proposing the need for verification mechanisms to improve AI-human collaboration in software development.

Contribution

We conducted a large-scale analysis of AI-generated PRs, identified common inconsistency types, and quantified their impact on acceptance rates and merge times, highlighting the importance of PR-MCI verification.

Findings

1. 1.7% of AI-generated PRs showed high message-code inconsistency.

2. High-MCI PRs had 51.7% lower acceptance rates.

3. High-MCI PRs took 3.5 times longer to merge.

Abstract

Pull request (PR) descriptions generated by AI coding agents are the primary channel for communicating code changes to human reviewers. However, the alignment between these messages and the actual changes remains unexplored, raising concerns about the trustworthiness of AI agents. To fill this gap, we analyzed 23,247 agentic PRs across five agents using PR message-code inconsistency (PR-MCI). We contributed 974 manually annotated PRs, found 406 PRs (1.7%) exhibited high PR-MCI, and identified eight PR-MCI types, revealing that "descriptions claim unimplemented changes" was the most common issue (45.4%). Statistical tests confirmed that high-MCI PRs had 51.7% lower acceptance rates (28.3% vs. 80.0%) and took 3.5 times longer to merge (55.8 vs. 16.0 hours). Our findings suggest that unreliable PR descriptions undermine trust in AI agents, highlighting the need for PR-MCI verification mechanisms and improved PR generation to enable trustworthy human-AI collaboration.