Studying Quality Improvements Recommended via Manual and Automated Code Review

TL;DR

This study compares human and AI (ChatGPT-4) code reviews, revealing AI's limitations in identifying quality issues but also its potential as a supplementary tool to human reviewers.

Contribution

It provides a detailed comparison of human and AI code review recommendations, highlighting the strengths and limitations of current DL-based approaches.

Findings

ChatGPT recommends 2.4 times more code changes than humans.

ChatGPT detects only 10% of issues identified by humans.

Approximately 40% of AI suggestions point to meaningful quality issues.

Abstract

Several Deep Learning (DL)-based techniques have been proposed to automate code review. Still, it is unclear the extent to which these approaches can recommend quality improvements as a human reviewer. We study the similarities and differences between code reviews performed by humans and those automatically generated by DL models, using ChatGPT-4 as representative of the latter. In particular, we run a mining-based study in which we collect and manually inspect 739 comments posted by human reviewers to suggest code changes in 240 PRs. The manual inspection aims at classifying the type of quality improvement recommended by human reviewers (e.g., rename variable/constant). Then, we ask ChatGPT to perform a code review on the same PRs and we compare the quality improvements it recommends against those suggested by the human reviewers. We show that while, on average, ChatGPT tends to recommend a higher number of code changes as compared to human reviewers (~2.4x more), it can only spot 10% of the quality issues reported by humans. However, ~40% of the additional comments generated by the LLM point to meaningful quality issues. In short, our findings show the complementarity of manual and AI-based code review. This finding suggests that, in its current state, DL-based code review can be used as a further quality check on top of the one performed by humans, but should not be considered as a valid alternative to them nor as a mean to save code review time, since human reviewers would still need to perform their manual inspection while also validating the quality issues reported by the DL-based technique.