LLMBisect: Breaking Barriers in Bug Bisection with A Comparative Analysis Pipeline

TL;DR

This paper introduces LLMBisect, a multi-stage pipeline leveraging large language models to improve bug bisection accuracy by fully utilizing patch information and comparing multiple candidates, surpassing existing methods.

Contribution

The paper presents a novel multi-stage LLM-based pipeline for bug bisection that overcomes limitations of traditional methods by integrating textual and code analysis.

Findings

Achieves over 38% higher accuracy than state-of-the-art solutions.

Improves accuracy by 60% over baseline LLM-based bisection.

Effectively utilizes patch information and candidate comparison in bug localization.

Abstract

Bug bisection has been an important security task that aims to understand the range of software versions impacted by a bug, i.e., identifying the commit that introduced the bug. However, traditional patch-based bisection methods are faced with several significant barriers: For example, they assume that the bug-inducing commit (BIC) and the patch commit modify the same functions, which is not always true. They often rely solely on code changes, while the commit message frequently contains a wealth of vulnerability-related information. They are also based on simple heuristics (e.g., assuming the BIC initializes lines deleted in the patch) and lack any logical analysis of the vulnerability. In this paper, we make the observation that Large Language Models (LLMs) are well-positioned to break the barriers of existing solutions, e.g., comprehend both textual data and code in patches and commits. Unlike previous BIC identification approaches, which yield poor results, we propose a comprehensive multi-stage pipeline that leverages LLMs to: (1) fully utilize patch information, (2) compare multiple candidate commits in context, and (3) progressively narrow down the candidates through a series of down-selection steps. In our evaluation, we demonstrate that our approach achieves significantly better accuracy than the state-of-the-art solution by more than 38\%. Our results further confirm that the comprehensive multi-stage pipeline is essential, as it improves accuracy by 60\% over a baseline LLM-based bisection method.