BugSpotter: Automated Generation of Code Debugging Exercises

TL;DR

BugSpotter is a tool that uses large language models to automatically generate debugging exercises from problem descriptions, helping students improve their debugging skills efficiently and effectively.

Contribution

This paper presents BugSpotter, a novel system that automatically creates debugging exercises using LLMs, reducing manual effort and maintaining quality comparable to instructor-crafted problems.

Findings

LLM-generated exercises matched instructor-crafted ones in difficulty.

Students performed similarly on LLM-generated and manual exercises.

BugSpotter effectively enhances debugging skill practice in classroom settings.

Abstract

Debugging is an essential skill when learning to program, yet its instruction and emphasis often vary widely across introductory courses. In the era of code-generating large language models (LLMs), the ability for students to reason about code and identify errors is increasingly important. However, students frequently resort to trial-and-error methods to resolve bugs without fully understanding the underlying issues. Developing the ability to identify and hypothesize the cause of bugs is crucial but can be time-consuming to teach effectively through traditional means. This paper introduces BugSpotter, an innovative tool that leverages an LLM to generate buggy code from a problem description and verify the synthesized bugs via a test suite. Students interact with BugSpotter by designing failing test cases, where the buggy code's output differs from the expected result as defined by the problem specification. This not only provides opportunities for students to enhance their debugging skills, but also to practice reading and understanding problem specifications. We deployed BugSpotter in a large classroom setting and compared the debugging exercises it generated to exercises hand-crafted by an instructor for the same problems. We found that the LLM-generated exercises produced by BugSpotter varied in difficulty and were well-matched to the problem specifications. Importantly, the LLM-generated exercises were comparable to those manually created by instructors with respect to student performance, suggesting that BugSpotter could be an effective and efficient aid for learning debugging.