A Bayesian Approach to Estimating Effect Sizes in Educational Research

TL;DR

This paper introduces a Bayesian method for estimating effect sizes in educational research, accounting for data structure and variance heterogeneity, implemented via R and Stan.

Contribution

It presents a novel Bayesian approach for effect size estimation in educational studies, incorporating multilevel data and heteroscedastic variances, with practical implementation guidance.

Findings

Provides a Bayesian framework for effect size estimation in educational data

Recommends specific effect size calculations for different study designs

Enables comparison of educational interventions across classes

Abstract

In this paper, we demonstrate a purely Bayesian approach for estimating within-group and between-group effect sizes for learning outcomes encountered in educational research, taking naturally into account the multilevel structure of the data, as well as heterogeneous residual variances among time points and conditions. We provide a detailed implementation using the brms package in R serving as a wrapper for the probabilistic programming language Stan. We recommend that for a pooled design, one computes an effect size $d_s$ similar to a Cohen's $d$, and for a paired design, one should compute two possibly different quantities $d_s$ and $d_z$ to correct for correlations in within-group designs and allow for comparability across different studies. All these effect sizes are based on ideas coming from Hedge's total effect size $\delta_t$ introduced in 2007. Ultimately, these estimates allow us to study the differential effectiveness of educational interventions with respect to classes.