The Right Kind of Help: Evaluating the Effectiveness of Intervention Methods in Elementary-Level Visual Programming

TL;DR

This large-scale study evaluates the effectiveness of different intervention methods in elementary programming, showing that interventions improve learning, engagement, and post-learning problem-solving skills, with quiz-based methods offering additional benefits.

Contribution

The paper provides a comprehensive comparison of intervention methods during and after learning in elementary programming, highlighting the effectiveness of quiz-based strategies.

Findings

All intervention methods improved learning performance.

Interventions increased student engagement and perceived skill growth.

Quiz-based interventions enhanced performance on novel tasks.

Abstract

Prior work has explored various intervention methods for elementary programming. However, the relative impact of these methods during the learning and post-learning phases remains unclear. In this work, we present a large-scale study comparing the effectiveness of various intervention methods in elementary programming both during learning and on novel tasks post-learning. Specifically, we compare three intervention methods: code-edit recommendations (Code-Rec), quizzes based on code edits (Code-Quiz), and quizzes based on metacognitive strategies (Plan-Quiz), along with a no-intervention control (group None). A total of 398 students (across grades 4-7) participated in a two-phase study: learning phase comprising write-code tasks from the Hour of Code: Maze Challenge with the intervention, followed by a post-learning phase comprising more advanced write-code tasks without any intervention. All intervention methods significantly improved learning performance over the control group while preserving students' problem-solving skills in the post-learning phase. Quiz-based methods further improved performance on novel post-learning tasks. Students in intervention groups also reported greater engagement and perceived skill growth.