Combining Logic with Large Language Models for Automatic Debugging and Repair of ASP Programs

TL;DR

This paper presents FormHe, a novel tool combining logic-based techniques and Large Language Models to automatically identify and repair faults in Answer Set Programming, aiding students in debugging complex logic programs.

Contribution

Introducing FormHe, the first system integrating logic-based fault localization with LLM-driven program repair for ASP programs.

Findings

94% fault detection accuracy on real student submissions

58% successful program repairs in experiments

Effective guidance for students in debugging ASP programs

Abstract

Logic programs are a powerful approach for solving NP-Hard problems. However, due to their declarative nature, debugging logic programs poses significant challenges. Unlike procedural paradigms, which allow for step-by-step inspection of program state, logic programs require reasoning about logical statements for fault localization. This complexity is amplified in learning environments due to students' inexperience. We introduce FormHe, a novel tool that combines logic-based techniques and Large Language Models to identify and correct issues in Answer Set Programming submissions. FormHe consists of two components: a fault localization module and a program repair module. First, the fault localizer identifies a set of faulty program statements requiring modification. Subsequently, FormHe employs program mutation techniques and Large Language Models to repair the flawed ASP program. These repairs can then serve as guidance for students to correct their programs. Our experiments with real buggy programs submitted by students show that FormHe accurately detects faults in 94% of cases and successfully repairs 58% of incorrect submissions.