Automated Code Review In Practice
Umut Cihan, Vahid Haratian, Arda \.I\c{c}\"oz, Mert Kaan G\"ul,, \"Omercan Devran, Emircan Furkan Bayendur, Baykal Mehmet U\c{c}ar, Eray, T\"uz\"un

TL;DR
This study evaluates the real-world impact of AI-assisted automated code review tools using large language models in industry, highlighting benefits in bug detection and code quality awareness, alongside challenges like increased review times and occasional inaccuracies.
Contribution
It provides an empirical analysis of LLM-based automated code review tools in an industrial setting, assessing their effects on developer workflows and code quality.
Findings
73.8% of automated comments were resolved
Automated reviews increased pull request closure times
Practitioners reported minor improvements in code quality
Abstract
Code review is a widespread practice to improve software quality and transfer knowledge. It is often seen as time-consuming due to the need for manual effort and potential delays. Several AI-assisted tools, such as Qodo, GitHub Copilot, and Coderabbit, provide automated reviews using large language models (LLMs). The effects of such tools in the industry are yet to be examined. This study examines the impact of LLM-based automated code review tools in an industrial setting. The study was conducted within a software development environment that adopted an AI-assisted review tool (based on open-source Qodo PR Agent). Around 238 practitioners across ten projects had access to the tool. We focused on three projects with 4,335 pull requests, 1,568 of which underwent automated reviews. Data collection comprised three sources: (1) a quantitative analysis of pull request data, including…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Testing and Debugging Techniques · Model-Driven Software Engineering Techniques · Advanced Software Engineering Methodologies
