Leveraging Reward Models for Guiding Code Review Comment Generation

TL;DR

This paper introduces CoRAL, a reinforcement learning-based framework that improves automated code review comment generation by rewarding comments that are semantically relevant and useful for code refinement, outperforming existing methods.

Contribution

The paper proposes a novel reinforcement learning approach for code review comment generation that considers comment usefulness and semantic similarity, enhancing the quality of automated reviews.

Findings

CoRAL generates more relevant and useful comments than baseline models.

The approach improves the semantic alignment of generated comments with expected reviews.

Quantitative and qualitative evaluations demonstrate the superiority of CoRAL.

Abstract

Code review is a crucial component of modern software development, involving the evaluation of code quality, providing feedback on potential issues, and refining the code to address identified problems. Despite these benefits, code review can be rather time consuming, and influenced by subjectivity and human factors. For these reasons, techniques to (partially) automate the code review process have been proposed in the literature. Among those, the ones exploiting deep learning (DL) are able to tackle the generative aspect of code review, by commenting on a given code as a human reviewer would do (i.e., comment generation task) or by automatically implementing code changes required to address a reviewer's comment (i.e., code refinement task). In this paper, we introduce CoRAL, a deep learning framework automating review comment generation by exploiting reinforcement learning with a reward mechanism considering both the semantics of the generated comments as well as their usefulness as input for other models automating the code refinement task. The core idea is that if the DL model generates comments that are semantically similar to the expected ones or can be successfully implemented by a second model specialized in code refinement, these comments are likely to be meaningful and useful, thus deserving a high reward in the reinforcement learning framework. We present both quantitative and qualitative comparisons between the comments generated by CoRAL and those produced by the latest baseline techniques, highlighting the effectiveness and superiority of our approach.