Who Writes the Docs in SE 3.0? Agent vs. Human Documentation Pull Requests

TL;DR

As AI agents increasingly contribute to software documentation in SE3.0, this study analyzes their role, highlighting their substantial contributions and raising concerns about review practices and documentation quality.

Contribution

This paper provides an empirical analysis of AI agent contributions to documentation in software projects, an area previously underexplored in SE3.0 research.

Findings

AI agents submit more documentation PRs than humans

Agent-authored documentation is integrated with minimal human modification

Concerns raised about review practices and documentation quality assurance

Abstract

As software engineering moves toward SE3.0, AI agents are increasingly used to carry out development tasks and contribute changes to software projects. It is therefore important to understand the extent of these contributions and how human developers review and intervene, since these factors shape the risks of delegating work to AI agents. While recent studies have examined how AI agents support software development tasks (e.g., code generation, issue resolution, and PR automation), their role in documentation tasks remains underexplored-even though documentation is widely consumed and shapes how developers understand and use software. Using the AIDev, we analyze 1,997 documentation-related pull requests (PRs) authored by AI agents and human developers, where documentation PRs are those that create or modify project documentation artifacts. We find that AI agents submit substantially more documentation-related PRs than humans in the studied repositories. We further observe that agent-authored documentation edits are typically integrated with little follow-up modification from humans, raising concerns about review practices and the reliability of agent-generated documentation. Overall, while AI agents already contribute substantially to documentation workflows, our results suggest concerns for emerging challenges for documentation quality assurance and human-AI collaboration in SE3.0.