A Robust, Differentially Private Randomized Experiment for Evaluating Online Educational Programs With Sensitive Student Data

TL;DR

This paper introduces a differentially private randomized experimental design for evaluating online educational programs that protects sensitive student data and remains robust against adversarial responses, providing unbiased treatment effect estimators.

Contribution

It presents a novel, privacy-preserving experimental design that is robust to noncompliance and adversarial responses, with new estimators for treatment effects in online education evaluations.

Findings

Successfully evaluated online statistics courses at UW-Madison during Spring 2021

Demonstrated robustness of the design against biased or contaminated responses

Provided unbiased, asymptotically normal estimators for treatment effects

Abstract

Randomized control trials (RCTs) have been the gold standard to evaluate the effectiveness of a program, policy, or treatment on an outcome of interest. However, many RCTs assume that study participants are willing to share their (potentially sensitive) data, specifically their response to treatment. This assumption, while trivial at first, is becoming difficult to satisfy in the modern era, especially in online settings where there are more regulations to protect individuals' data. The paper presents a new, simple experimental design that is differentially private, one of the strongest notions of data privacy. Also, using works on noncompliance in experimental psychology, we show that our design is robust against "adversarial" participants who may distrust investigators with their personal data and provide contaminated responses to intentionally bias the results of the experiment. Under our new design, we propose unbiased and asymptotically Normal estimators for the average treatment effect. We also present a doubly robust, covariate-adjusted estimator that uses pre-treatment covariates (if available) to improve efficiency. We conclude by using the proposed experimental design to evaluate the effectiveness of online statistics courses at the University of Wisconsin-Madison during the Spring 2021 semester, where many classes were online due to COVID-19.