Using Adaptive Experiments to Rapidly Help Students

TL;DR

This paper demonstrates how adaptive experiments, specifically using Thompson Sampling, can efficiently identify effective educational interventions, exemplified by a case study on email reminders for students.

Contribution

It provides a practical case study of implementing adaptive experiments in education and discusses their advantages and limitations compared to traditional methods.

Findings

Adaptive experiments can more quickly identify effective interventions.

Thompson Sampling outperforms uniform random assignment in this context.

The paper raises open questions about the broader applicability of adaptive experiments.

Abstract

Adaptive experiments can increase the chance that current students obtain better outcomes from a field experiment of an instructional intervention. In such experiments, the probability of assigning students to conditions changes while more data is being collected, so students can be assigned to interventions that are likely to perform better. Digital educational environments lower the barrier to conducting such adaptive experiments, but they are rarely applied in education. One reason might be that researchers have access to few real-world case studies that illustrate the advantages and disadvantages of these experiments in a specific context. We evaluate the effect of homework email reminders in students by conducting an adaptive experiment using the Thompson Sampling algorithm and compare it to a traditional uniform random experiment. We present this as a case study on how to conduct such experiments, and we raise a range of open questions about the conditions under which adaptive randomized experiments may be more or less useful.