Debugging with Open-Source Large Language Models: An Evaluation

TL;DR

This paper evaluates the effectiveness of open-source large language models in debugging buggy code across multiple programming languages, providing insights into their capabilities and limitations.

Contribution

It offers the first comprehensive evaluation of open-source LLMs' performance in debugging, using a large benchmark across three programming languages.

Findings

Open-source LLMs achieved 43.9% to 66.6% success rates.

DeepSeek-Coder performed best across all languages.

Evaluation covers over 4000 buggy code instances.

Abstract

Large language models have shown good potential in supporting software development tasks. This is why more and more developers turn to LLMs (e.g. ChatGPT) to support them in fixing their buggy code. While this can save time and effort, many companies prohibit it due to strict code sharing policies. To address this, companies can run open-source LLMs locally. But until now there is not much research evaluating the performance of open-source large language models in debugging. This work is a preliminary evaluation of the capabilities of open-source LLMs in fixing buggy code. The evaluation covers five open-source large language models and uses the benchmark DebugBench which includes more than 4000 buggy code instances written in Python, Java and C++. Open-source LLMs achieved scores ranging from 43.9% to 66.6% with DeepSeek-Coder achieving the best score for all three programming languages.