Measuring the Impact of Distractors on Student Learning Gains while Using Proof Blocks

TL;DR

This study investigates how distractors in Proof Blocks activities influence student learning gains, finding that distractors may improve performance, but results were not statistically significant, highlighting the need for further research.

Contribution

It is the first experimental study to compare learning gains from Proof Blocks activities with and without distractors using a randomized controlled trial.

Findings

Proof Blocks with distractors led to higher posttest scores than without distractors.

No statistically significant differences were found among the groups.

Results suggest potential benefits of distractors, but further research is needed.

Abstract

Background: Proof Blocks is a software tool that enables students to construct proofs by assembling prewritten lines and gives them automated feedback. Prior work on learning gains from Proof Blocks has focused on comparing learning gains from Proof Blocks against other learning activities such as writing proofs or reading. Purpose: The study described in this paper aims to compare learning gains from different variations of Proof Blocks. Specifically, we attempt to quantify the difference in learning gains for students who complete Proof Blocks problems with and without distractors. Methods: We conducted a randomized controlled trial with three experimental groups: a control group that completed an off-topic Proof Blocks activity, one that completed a \tool{} activity without distractors, and one that completed a Proof Blocks activity with distractors. All three groups read a book chapter on proof by induction before completing their activity. Findings: The group that completed the Proof Blocks activity with distractors performed better on the posttest than the group that completed the Proof Blocks without distractors, who in turn performed better than the group that completed the off-topic Proof Blocks activity. However, none of these differences were statistically significant. While the results of this study are inconclusive, we hope that it can serve as a foundation for future work.