Effects of Hints on Debugging Scratch Programs: An Empirical Study with Primary School Teachers in Training

TL;DR

This study evaluates how automatically generated hints on bug patterns affect primary school teachers in training when debugging Scratch programs, showing reduced effort and increased correctness, but no significant impact on learning outcomes.

Contribution

It demonstrates that hints on bug patterns can improve debugging efficiency and effectiveness, and explores their influence on teachers' learning process.

Findings

Hints reduce debugging time from 8.66 to 5.24 minutes.

Hints increase correct solutions by 34%.

No significant difference in learning effects between training with or without hints.

Abstract

Bugs in learners' programs are often the result of fundamental misconceptions. Teachers frequently face the challenge of first having to understand such bugs, and then suggest ways to fix them. In order to enable teachers to do so effectively and efficiently, it is desirable to support them in recognising and fixing bugs. Misconceptions often lead to recurring patterns of similar bugs, enabling automated tools to provide this support in terms of hints on occurrences of common bug patterns. In this paper, we investigate to what extent the hints improve the effectiveness and efficiency of teachers in debugging learners' programs using a cohort of 163 primary school teachers in training, tasked to correct buggy Scratch programs, with and without hints on bug patterns. Our experiment suggests that automatically generated hints can reduce the effort of finding and fixing bugs from 8.66 to 5.24 minutes, while increasing the effectiveness by 34% more correct solutions. While this improvement is convincing, arguably teachers in training might first need to learn debugging "the hard way" to not miss the opportunity to learn by relying on tools. We therefore investigate whether the use of hints during training affects their ability to recognise and fix bugs without hints. Our experiment provides no significant evidence that either learning to debug with hints or learning to debug "the hard way" leads to better learning effects. Overall, this suggests that bug patterns might be a useful concept to include in the curriculum for teachers in training, while tool-support to recognise these patterns is desirable for teachers in practice.