How can Email Interventions Increase Students' Completion of Online Homework? A Case Study Using A/B Comparisons

TL;DR

This study uses randomized A/B email experiments to evaluate how different email designs influence students' motivation and timing in completing online homework, providing empirical insights for improving email interventions.

Contribution

It introduces an experimental paradigm for testing email intervention designs using A/B comparisons and offers qualitative insights into student and instructor perceptions.

Findings

Some email designs motivate earlier homework attempts

Predictions about email effectiveness often do not match actual impact

Qualitative data reveals student opinions and behaviors post-intervention

Abstract

Email communication between instructors and students is ubiquitous, and it could be valuable to explore ways of testing out how to make email messages more impactful. This paper explores the design space of using emails to get students to plan and reflect on starting weekly homework earlier. We deployed a series of email reminders using randomized A/B comparisons to test alternative factors in the design of these emails, providing examples of an experimental paradigm and metrics for a broader range of interventions. We also surveyed and interviewed instructors and students to compare their predictions about the effectiveness of the reminders with their actual impact. We present our results on which seemingly obvious predictions about effective emails are not borne out, despite there being evidence for further exploring these interventions, as they can sometimes motivate students to attempt their homework more often. We also present qualitative evidence about student opinions and behaviours after receiving the emails, to guide further interventions. These findings provide insight into how to use randomized A/B comparisons in everyday channels such as emails, to provide empirical evidence to test our beliefs about the effectiveness of alternative design choices.