Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness

TL;DR

This paper investigates how different prompting strategies for GPT-3.5 affect the quality of programming error explanations generated without original error messages, aiming to improve novice learning and debugging.

Contribution

It explores the impact of various prompting techniques on LLM-generated error explanations using only erroneous code, without original error messages, which is a novel approach.

Findings

Prompting strategies influence explanation quality

Fine-tuning improves error explanation effectiveness

One-shot prompts outperform baseline methods

Abstract

Making errors is part of the programming process -- even for the most seasoned professionals. Novices in particular are bound to make many errors while learning. It is well known that traditional (compiler/interpreter) programming error messages have been less than helpful for many novices and can have effects such as being frustrating, containing confusing jargon, and being downright misleading. Recent work has found that large language models (LLMs) can generate excellent error explanations, but that the effectiveness of these error messages heavily depends on whether the LLM has been provided with context -- typically the original source code where the problem occurred. Knowing that programming error messages can be misleading and/or contain that serves little-to-no use (particularly for novices) we explore the reverse: what happens when GPT-3.5 is prompted for error explanations on just the erroneous source code itself -- original compiler/interpreter produced error message excluded. We utilized various strategies to make more effective error explanations, including one-shot prompting and fine-tuning. We report the baseline results of how effective the error explanations are at providing feedback, as well as how various prompting strategies might improve the explanations' effectiveness. Our results can help educators by understanding how LLMs respond to such prompts that novices are bound to make, and hopefully lead to more effective use of Generative AI in the classroom.