COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis

TL;DR

This paper introduces DEBUGEVAL, a comprehensive benchmark for evaluating LLMs' multi-stage code debugging abilities, and proposes COAST, a multi-agent data synthesis framework that significantly improves the debugging performance of smaller models.

Contribution

The paper presents DEBUGEVAL for multi-stage debugging evaluation and introduces COAST, a novel multi-agent data synthesis method that enhances small LLMs' debugging capabilities.

Findings

7B-scale models underperform larger models in debugging tasks

COAST-generated data improves small LLM debugging performance

COAST enables 7B models to match GPT-3.5 debugging abilities

Abstract

Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human debugging typically follows a multi-stage process, which includes Bug Localization, Bug Identification, Code Repair, and Code Recognition. However, existing code debugging benchmarks predominantly focus on the Code Repair stage, which offers only a limited perspective on evaluating the debugging capabilities of LLMs. In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process. Through evaluating on DEBUGEVAL, we observe that 7B-scale models consistently underperform compared to their larger counterparts, highlighting their limitations in comprehending code semantics. In this case, we propose the COmmunicative Agent-based data SynThesis (COAST) framework, which employs a multi-agent system to generate high-quality training data for supervised fine-tuning (SFT). Experimental results demonstrate that COAST-generated data outperform human-curated and GPT-4-generated data, enabling 7B-scale LLMs to achieve debugging performance comparable to GPT-3.5. All data and codes are available at https://github.com/NEUIR/COAST.