Effects of Automated Interventions in Programming Assignments: Evidence from a Field Experiment

TL;DR

This study investigates automated interventions in MOOCs, demonstrating that timely prompts and personalized exercises can significantly increase help-seeking behavior and reduce struggle time, thereby enhancing learning support.

Contribution

The paper introduces adaptive automatic interventions that encourage help requests and provide personalized exercises, improving support for struggling students in large-scale online courses.

Findings

Help call-outs increased by up to 66%

Reduced time students spent before seeking help

Insights for course material improvement

Abstract

A typical problem in MOOCs is the missing opportunity for course conductors to individually support students in overcoming their problems and misconceptions. This paper presents the results of automatically intervening on struggling students during programming exercises and offering peer feedback and tailored bonus exercises. To improve learning success, we do not want to abolish instructionally desired trial and error but reduce extensive struggle and demotivation. Therefore, we developed adaptive automatic just-in-time interventions to encourage students to ask for help if they require considerably more than average working time to solve an exercise. Additionally, we offered students bonus exercises tailored for their individual weaknesses. The approach was evaluated within a live course with over 5,000 active students via a survey and metrics gathered alongside. Results show that we can increase the call outs for help by up to 66% and lower the dwelling time until issuing action. Learnings from the experiments can further be used to pinpoint course material to be improved and tailor content to be audience specific.