Do AI Agents Really Improve Code Readability?

TL;DR

This paper investigates whether AI agent-based refactoring improves code readability, analyzing 403 commits and finding that while AI targets complexity and documentation, it often degrades traditional quality metrics.

Contribution

It provides an empirical analysis of AI agent refactoring's impact on code readability, focusing specifically on readability-related changes and their effects on quality metrics.

Findings

AI refactoring mainly targets logic complexity and documentation.

Readability-focused commits often decrease maintainability and increase complexity.

AI refactoring does not consistently improve traditional code quality metrics.

Abstract

Code readability is fundamental to software quality and maintainability. Poor readability extends development time, increases bug-inducing risks, and contributes to technical debt. With the rapid advancement of Large Language Models, AI agent-based approaches have emerged as a promising paradigm for automated refactoring, capable of decomposing complex tasks through autonomous planning and execution. While prior studies have examined refactoring by AI agents, these analyses cover all forms of refactoring, including performance optimization and structural improvement. As a result, the extent to which AI agent-based refactoring specifically improves code readability remains unclear. This study investigates the impact of AI agent-based refactoring on code readability. We extracted commits containing readability-related keywords from the AIDev dataset and analyzed changes in readability metrics before and after each commit, covering 403 commits evaluated using multiple quantitative metrics. Our results indicate that AI agents primarily target logic complexity (42.4%) and documentation improvements (24.2%) rather than surface-level aspects like naming conventions or formatting. However, contrary to expectations, readability-focused commits often degraded traditional quality metrics: the Maintainability Index decreased in 56.1% of commits, while Cyclomatic Complexity increased in 42.7%.