Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study

TL;DR

This study evaluates open-source Llama models, especially CodeLlama, for automated code refinement, demonstrating their potential as cost-effective and privacy-preserving alternatives to proprietary models like ChatGPT in software development.

Contribution

It provides an empirical comparison showing that properly tuned Llama models can perform comparably to ChatGPT in code refinement tasks, especially in code modification scenarios.

Findings

CodeLlama achieves performance close to ChatGPT in code refinement.

Refactoring tasks are more effectively automated than generating new code.

Open-source models offer privacy and cost advantages for real-world applications.

Abstract

Code reviews are an integral part of software development and have been recognized as a crucial practice for minimizing bugs and favouring higher code quality. They serve as an important checkpoint before committing code and play an essential role in knowledge transfer between developers. However, code reviews can be time-consuming and can stale the development of large software projects. In a recent study, Guo et al. assessed how ChatGPT3.5 can help the code review process. They evaluated the effectiveness of ChatGPT in automating the code refinement tasks, where developers recommend small changes in the submitted code. While Guo et al. 's study showed promising results, proprietary models like ChatGPT pose risks to data privacy and incur extra costs for software projects. In this study, we explore alternatives to ChatGPT in code refinement tasks by including two open-source, smaller-scale large language models: CodeLlama and Llama 2 (7B parameters). Our results show that, if properly tuned, the Llama models, particularly CodeLlama, can achieve reasonable performance, often comparable to ChatGPT in automated code refinement. However, not all code refinement tasks are equally successful: tasks that require changing existing code (e.g., refactoring) are more manageable for models to automate than tasks that demand new code. Our study highlights the potential of open-source models for code refinement, offering cost-effective, privacy-conscious solutions for real-world software development.