How Small is Enough? Empirical Evidence of Quantized Small Language Models for Automated Program Repair

TL;DR

Small language models, when quantized, can match or surpass large models in automated program repair tasks, offering a resource-efficient alternative with minimal accuracy loss.

Contribution

This study demonstrates that small language models, especially with int8 quantization, can achieve competitive bug-fixing performance compared to large models in APR.

Findings

SLMs can fix bugs as accurately as LLMs.

Int8 quantization minimally impacts APR accuracy.

Quantized SLMs significantly reduce memory usage.

Abstract

Background: Large language models (LLMs) have greatly improved the accuracy of automated program repair (APR) methods. However, LLMs are constrained by high computational resource requirements. Aims: We focus on small language models (SLMs), which perform well even with limited computational resources compared to LLMs. We aim to evaluate whether SLMs can achieve competitive performance in APR tasks. Method: We conducted experiments on the QuixBugs benchmark to compare the bug-fixing accuracy of SLMs and LLMs. We also analyzed the impact of int8 quantization on APR performance. Results: The latest SLMs can fix bugs as accurately as--or even more accurately than--LLMs. Also, int8 quantization had minimal effect on APR accuracy while significantly reducing memory requirements. Conclusions: SLMs present a viable alternative to LLMs for APR, offering competitive accuracy with lower computational costs, and quantization can further enhance their efficiency without compromising effectiveness.