Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks

TL;DR

This paper introduces BICS, a benchmark to evaluate LLMs' ability to detect syntax bugs in large Python code stacks, revealing challenges and disparities in model performance.

Contribution

The paper presents BICS, a novel benchmark for assessing LLMs' bug detection in large codebases, highlighting performance issues related to context length and model differences.

Findings

Code environments are more challenging than text environments for retrieval.

Significant performance differences exist among models.

Longer context lengths generally degrade model performance.

Abstract

Recent research in Needle-in-a-Haystack (NIAH) benchmarks has explored the capabilities of Large Language Models (LLMs) in retrieving contextual information from large text documents. However, as LLMs become increasingly integrated into software development processes, it is crucial to evaluate their performance in code-based environments. As LLMs are further developed for program synthesis, we need to ensure that LLMs can understand syntax and write syntactically correct code. As a step in ensuring LLMs understand syntax, LLMs can be evaluated in their ability to find and detect syntax bugs. Our benchmark, Bug In The Code Stack (BICS), is designed to assess the ability of LLMs to identify simple syntax bugs within large source code. Our findings reveal three key insights: (1) code-based environments pose significantly more challenge compared to text-based environments for retrieval tasks, (2) there is a substantial performance disparity among different models, and (3) there is a notable correlation between longer context lengths and performance degradation, though the extent of this degradation varies between models.