Exploring the Role of Tracing in AI-Supported Planning for Algorithmic Reasoning

TL;DR

This study investigates how explicit tracing in AI-supported planning tools influences algorithmic reasoning, showing it encourages goal-driven thinking and more consistent solutions without affecting feedback quality.

Contribution

It provides empirical evidence on the effects of integrating explicit tracing into AI-assisted programming education, highlighting its impact on reasoning and solution consistency.

Findings

Tracing shifts reasoning from line-by-line to goal-driven approaches

Tracing leads to more consistent partially correct solutions

Tracing does not significantly change feedback quality or reliability

Abstract

AI-powered planning tools show promise in supporting programming learners by enabling early, formative feedback on their thinking processes prior to coding. To date, however, most AI-supported planning tools rely on students' natural-language explanations, using LLMs to interpret learners' descriptions of their algorithmic intent. Prior to the emergence of LLM-based systems, CS education research extensively studied trace-based planning in pen-and-paper settings, demonstrating that reasoning through stepwise execution with explicit state transitions helps learners build and refine mental models of program behavior. Despite its potential, little is known about how tracing interacts with AI-mediated feedback and whether integrating tracing into AI-supported planning tools leads to different learning processes or interaction dynamics compared to natural-language-based planning alone. We study how requiring learners to produce explicit execution traces with an AI-supported planning tool affects their algorithmic reasoning. In a between-subjects study with 20 students, tracing shifted learners away from code-like, line-by-line descriptions toward more goal-driven reasoning about program behavior. Moreover, it led to more consistent partially correct solutions, although final coding performance remained comparable across conditions. Notably, tracing did not significantly affect the quality or reliability of LLM-generated feedback. These findings reveal tradeoffs in combining tracing with AI-supported planning and inform design guidelines for integrating natural language, tracing, and coding to support iterative reasoning throughout the programming process.