Experimental designs for multiple-level responses, with application to a large-scale educational intervention

TL;DR

This paper compares three experimental designs for educational interventions involving clustered subjects, analyzing their efficiency and power at multiple levels through simulations and providing a tool for study design optimization.

Contribution

It introduces a comparative analysis of three experimental designs for multi-level responses in clustered educational settings and offers a computational tool for optimizing design choices.

Findings

Randomizing schools reduces contamination but may lower power.

Randomizing teachers within schools balances power and contamination.

Simulation results guide optimal design selection based on correlation levels.

Abstract

Educational research often studies subjects that are in naturally clustered groups of classrooms or schools. When designing a randomized experiment to evaluate an intervention directed at teachers, but with effects on teachers and their students, the power or anticipated variance for the treatment effect needs to be examined at both levels. If the treatment is applied to clusters, power is usually reduced. At the same time, a cluster design decreases the probability of contamination, and contamination can also reduce power to detect a treatment effect. Designs that are optimal at one level may be inefficient for estimating the treatment effect at another level. In this paper we study the efficiency of three designs and their ability to detect a treatment effect: randomize schools to treatment, randomize teachers within schools to treatment, and completely randomize teachers to treatment. The three designs are compared for both the teacher and student level within the mixed model framework, and a simulation study is conducted to compare expected treatment variances for the three designs with various levels of correlation within and between clusters. We present a computer program that study designers can use to explore the anticipated variances of treatment effects under proposed experimental designs and settings.