Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts

TL;DR

This paper enhances large language models' ability to classify logical programming errors by incorporating error relationships into prompts, significantly improving accuracy and creating a new dataset for logical errors.

Contribution

It introduces a novel prompt-based method leveraging error relationships for better logical error classification and develops a new logical error dataset using LLMs.

Findings

21% improvement in classification accuracy with error relationship prompts

Effective use of Chain-of-Thought and Tree-of-Thought prompts

Creation of a new logical error dataset for programming applications

Abstract

LLMs trained in the understanding of programming syntax are now providing effective assistance to developers and are being used in programming education such as in generation of coding problem examples or providing code explanations. A key aspect of programming education is understanding and dealing with error message. However, 'logical errors' in which the program operates against the programmer's intentions do not receive error messages from the compiler. In this study, building on existing research on programming errors, we first define the types of logical errors that can occur in programming in general. Based on the definition, we propose an effective approach for detecting logical errors with LLMs that makes use of relations among error types in the Chain-of-Thought and Tree-of-Thought prompts. The experimental results indicate that when such logical error descriptions in the prompt are used, the average classifition performance is about 21% higher than the ones without them. We also conducted an experiment for exploiting the relations among errors in generating a new logical error dataset using LLMs. As there is very limited dataset for logical errors such benchmark dataset can be very useful for various programming related applications. We expect that our work can assist novice programmers in identifying the causes of code errors and correct them more effectively.