Rethinking Code Review Workflows with LLM Assistance: An Empirical Study

TL;DR

This study investigates how large language models can assist in code reviews by identifying challenges in traditional practices and testing two LLM-based tools, showing AI-led reviews are generally preferred but depend on context.

Contribution

It provides empirical insights into integrating LLMs into code review workflows and compares two prototype approaches for enhancing review quality and efficiency.

Findings

AI-led reviews are generally preferred by developers.

Effectiveness of LLM assistance depends on reviewer familiarity and pull request severity.

Challenges include trust issues and false positives in LLM-generated reviews.

Abstract

Code reviews are a critical yet time-consuming aspect of modern software development, increasingly challenged by growing system complexity and the demand for faster delivery. This paper presents a study conducted at WirelessCar Sweden AB, combining an exploratory field study of current code review practices with a field experiment involving two variations of an LLM-assisted code review tool. The field study identifies key challenges in traditional code reviews, including frequent context switching, insufficient contextual information, and highlights both opportunities (e.g., automatic summarization of complex pull requests) and concerns (e.g., false positives and trust issues) in using LLMs. In the field experiment, we developed two prototype variations: one offering LLM-generated reviews upfront and the other enabling on-demand interaction. Both utilize a semantic search pipeline based on retrieval-augmented generation to assemble relevant contextual information for the review, thereby tackling the uncovered challenges. Developers evaluated both variations in real-world settings: AI-led reviews are overall more preferred, while still being conditional on the reviewers' familiarity with the code base, as well as on the severity of the pull request.