Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study

TL;DR

This study evaluates the acceptance and impact of LLM-generated review comments in real-world software review settings across open and closed-source organizations, showing moderate acceptance and comparable effectiveness to human comments.

Contribution

It provides the first large-scale empirical user study on LLM-generated review comments in live environments, demonstrating their acceptance and influence on the review process.

Findings

8.1% and 7.2% of generated comments accepted by reviewers

Refactoring comments more likely to be accepted than functional ones

Generated comments are as effective as human comments in prompting future revisions

Abstract

We conduct a large-scale empirical user study in a live setup to evaluate the acceptance of LLM-generated comments and their impact on the review process. This user study was performed in two organizations, Mozilla (which has its codebase available as open source) and Ubisoft (fully closed-source). Inside their usual review environment, participants were given access to RevMate, an LLM-based assistive tool suggesting generated review comments using an off-the-shelf LLM with Retrieval Augmented Generation to provide extra code and review context, combined with LLM-as-a-Judge, to auto-evaluate the generated comments and discard irrelevant cases. Based on more than 587 patch reviews provided by RevMate, we observed that 8.1% and 7.2%, respectively, of LLM-generated comments were accepted by reviewers in each organization, while 14.6% and 20.5% other comments were still marked as valuable as review or development tips. Refactoring-related comments are more likely to be accepted than Functional comments (18.2% and 18.6% compared to 4.8% and 5.2%). The extra time spent by reviewers to inspect generated comments or edit accepted ones (36/119), yielding an overall median of 43s per patch, is reasonable. The accepted generated comments are as likely to yield future revisions of the revised patch as human-written comments (74% vs 73% at chunk-level).