LLM Chatbots in High School Programming: Exploring Behaviors and Interventions

TL;DR

This study investigates how guiding high school students in their use of LLM chatbots affects their help-seeking behaviors and learning outcomes, highlighting the importance of pedagogy in effective tool integration.

Contribution

It introduces a pedagogical intervention to teach instrumental help-seeking with LLMs and evaluates its impact through a Design-Based Research cycle.

Findings

Behavioral change in help-seeking patterns after intervention

No significant improvement in exam grades despite behavioral shifts

Guided help-seeking influences learning workflows but not necessarily performance

Abstract

This study uses a Design-Based Research (DBR) cycle to refine the integration of Large Language Models (LLMs) in high school programming education. The initial problem was identified in an Intervention Group where, in an unguided setting, a higher proportion of executive, solution-seeking queries correlated strongly and negatively with exam performance. A contemporaneous Comparison Group demonstrated that without guidance, these unproductive help-seeking patterns do not self-correct, with engagement fluctuating and eventually declining. This insight prompted a mid-course pedagogical intervention in the first group, designed to teach instrumental help-seeking. The subsequent evaluation confirmed the intervention's success, revealing a decrease in executive queries, as well as a shift toward more productive learning workflows. However, this behavioral change did not translate into a statistically significant improvement in exam grades, suggesting that altering tool-use strategies alone may be insufficient to overcome foundational knowledge gaps. The DBR process thus yields a more nuanced principle: the educational value of an LLM depends on a pedagogy that scaffolds help-seeking, but this is only one part of the complex process of learning.