Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials

TL;DR

This paper introduces a novel method using ecometric data to identify sources of impact variability across sites in multi-site social intervention trials, addressing confounding and measurement error.

Contribution

The paper proposes a new approach leveraging aggregate participant reports to explore impact variance, with theoretical validation and practical applications.

Findings

Method effectively identifies impact sources in simulated data.

Application demonstrates utility in real-world multi-site trials.

Addresses measurement error bias in impact assessment.

Abstract

A new method is proposed to explore sources of cross-site impact variance in multi-site trials of social interventions. With this approach, aggregate reports from participants in the treatment arm about the treatment experience are used to define potential mediators and aggregate reports from participants in the control arm about the outcome of interest are used to remove confounding due to unmeasured local contextual factors. Particular attention is paid to correcting bias due to measurement error in the mediator. Asymptotic theory for the validity of this approach is provided. Two example applications are given, one simulated and one empirical.