An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code

TL;DR

This study systematically evaluates the effectiveness of locally deployed LLMs, LLaMA 3.2 and Mistral, in detecting Python bugs across multiple projects, revealing moderate accuracy and highlighting challenges in bug localization.

Contribution

It provides the first comprehensive empirical analysis of local LLMs for bug detection in Python, using a large benchmark and automated evaluation framework.

Findings

Locally deployed models achieve 43-45% accuracy in bug detection.

A large proportion of responses are partially correct, indicating some bug localization.

Performance varies significantly across different projects and code characteristics.

Abstract

Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for real-world Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function level and an automated keyword-based evaluation framework. Our results show that locally executed models achieve accuracy between 43% and 45%, while producing a large proportion of partially correct responses that identify problematic code regions without pinpointing the exact fix. Performance varies significantly across projects, highlighting the importance of codebase characteristics. The results demonstrate that local models can identify a meaningful share of bugs, though precise localization remains difficult for locally executed LLMs, particularly when handling complex and context dependent bugs in realistic development scenarios.