Automating Patch Set Generation from Code Review Comments Using Large Language Models

TL;DR

This paper evaluates the effectiveness of large language models in automatically generating code patches from review comments, aiming to support developers and improve code review efficiency.

Contribution

It is the first study to compare multiple LLMs in generating code patches from review comments, providing insights into their current capabilities and future potential.

Findings

LLMs can generate relevant code patches based on review comments.

Performance varies significantly across different LLMs.

The study highlights the potential for automating parts of the code review process.

Abstract

The advent of Large Language Models (LLMs) has revolutionized various domains of artificial intelligence, including the realm of software engineering. In this research, we evaluate the efficacy of pre-trained LLMs in replicating the tasks traditionally performed by developers in response to code review comments. We provide code contexts to five popular LLMs and obtain the suggested code-changes (patch sets) derived from real-world code-review comments. The performance of each model is meticulously assessed by comparing their generated patch sets against the historical data of human-generated patch-sets from the same repositories. This comparative analysis aims to determine the accuracy, relevance, and depth of the LLMs' feedback, thereby evaluating their readiness to support developers in responding to code-review comments. Novelty: This particular research area is still immature requiring a substantial amount of studies yet to be done. No prior research has compared the performance of existing Large Language Models (LLMs) in code-review comments. This in-progress study assesses current LLMs in code review and paves the way for future advancements in automated code quality assurance, reducing context-switching overhead due to interruptions from code change requests.