Reasoning Trajectories for Socratic Debugging of Student Code: From Misconceptions to Contradictions and Updated Beliefs

TL;DR

This paper introduces reasoning trajectories for Socratic debugging, using large language models to generate and evaluate guided reasoning paths that help students identify misconceptions in code.

Contribution

It formulates reasoning trajectory generation as a new task, creates a dataset, and demonstrates LLM-based solutions with high accuracy in generating correct trajectories and conversations.

Findings

LLMs can generate up to 91% correct reasoning trajectories.

LLMs achieve 98.7% validity in conversation turns.

The dataset includes manually and automatically annotated debugging problems.

Abstract

In Socratic debugging, instructors guide students towards identifying and fixing a bug on their own, instead of providing the bug fix directly. Most novice programmer bugs are caused by programming misconceptions, namely false beliefs about a programming concept. In this context, Socratic debugging can be formulated as a guided Reasoning Trajectory (RT) leading to a statement about the program behavior that contradicts the bug-causing misconception. Upon reaching this contradiction, the ensuing cognitive dissonance is expected to lead the student to identify the false belief on their own, followed by an enduring belief update. In this paper, we introduce the task of reasoning trajectory generation, together with a dataset of debugging problems annotated with RTs that are manually created or LLM-generated. We then describe LLM-based solutions for generating RTs and Socratic conversations that are anchored on them. A large-scale LLM-as-judge evaluation shows that large language and reasoning models can generate up to 91% correct reasoning trajectories and 98.7% valid conversation turns.