Exploring the Capabilities of LLMs for Code Change Related Tasks

TL;DR

This paper empirically evaluates large language models on code-change tasks like review and commit message generation, revealing their strengths, limitations, and the impact of tuning methods and model size.

Contribution

It provides a comprehensive analysis of LLMs' performance on code change tasks, highlighting the effects of in-context learning and parameter-efficient fine-tuning.

Findings

LLMs perform better with examples but more examples don't always help.

LoRA-tuned LLMs are comparable to small pre-trained models.

Llama 2 and Code Llama models are consistently the best performers.

Abstract

Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks. To answer this question, we conduct an empirical study using \textgreater 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient fine-tuning (PEFT, including LoRA and prefix-tuning). We observe that the performance of LLMs is poor without examples and generally improves with examples, but more examples do not always lead to better performance. LLMs tuned with LoRA have comparable performance to the state-of-the-art small pre-trained models. Larger models are not always better, but \textsc{Llama~2} and \textsc{Code~Llama} families are always the best. The best LLMs outperform small pre-trained models on the code changes that only modify comments and perform comparably on other code changes. We suggest future work should focus more on guiding LLMs to learn the knowledge specific to the changes related to code rather than comments for code-change-related tasks.