Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step

TL;DR

This paper introduces LDB, a framework that enables large language models to debug code by analyzing runtime execution step-by-step, improving their ability to identify and fix errors in generated programs.

Contribution

LDB uniquely segments programs into basic blocks and tracks intermediate variables, allowing LLMs to verify and debug code more effectively using runtime information.

Findings

LDB improves debugging performance by up to 9.8%.

Achieves state-of-the-art results on multiple benchmarks.

Enhances LLMs' ability to identify errors in complex code flows.

Abstract

Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated programs as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments the programs into basic blocks and tracks the values of intermediate variables after each block throughout the runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and efficiently pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.